What is the attention mechanism?
Before attention, sequence models had to compress everything they had seen into a single fixed-size vector before producing output. For short sequences, this worked. For long ones, it failed. Attention replaced that bottleneck with a mechanism that allows every output to look back at every input directly.
Definition
The attention mechanism is a neural network component that computes a weighted relationship between elements of a sequence, allowing each element to attend to every other element and aggregate information based on relevance.
For a given output position, attention asks: which other positions in the input are most relevant to producing this output? It answers that question by computing similarity scores between the current position and all others, converting those scores to weights, and taking a weighted sum of the values at all positions. The result is a context-aware representation that incorporates information from across the entire sequence.
How attention works
Attention operates on three vectors derived from each position in the sequence: a query (Q), a key (K), and a value (V).
For each position i, the attention score with position j is computed as the dot product of Q_i and K_j, scaled by the square root of the dimension. These scores are softmaxed to produce weights that sum to 1. The output for position i is then the weighted sum of all values V_j.
The query represents "what am I looking for?" The key represents "what do I contain?" The value represents "what do I contribute?" High similarity between a query and a key produces a high attention weight, meaning the value at that position contributes heavily to the output.
In multi-head attention, this operation runs in parallel across multiple independent "heads," each with different learned Q, K, and V projections. Different heads learn to attend to different types of relationships simultaneously.
Self-attention vs. cross-attention
Self-attention computes attention within a single sequence. Each position attends to every other position in the same input. This is how a transformer builds context: each token in a sentence attends to every other token, learning their relationships.
Cross-attention computes attention between two different sequences. For example, a generation model attends to a text prompt encoding while generating video frames. The query comes from the generation, the keys and values come from the conditioning signal. This is how conditioning information is injected into the generation process.
Attention in video generation
For video, attention must operate across two dimensions: space (within each frame) and time (across frames).
Spatial attention attends within a single frame, learning relationships between regions of the image: how the sky relates to the ground, where the subject is relative to the background.
Temporal attention attends across frames, learning how each region of the video evolves over time. This is what enables temporal consistency: the model can see how a feature looked in earlier frames and maintain it in later ones.
Spatiotemporal attention handles both simultaneously in a single attention operation. Rather than separating spatial and temporal attention into different stages, spatiotemporal attention allows the model to reason about how spatial regions in one frame relate to spatial regions in other frames directly. This is more powerful but more computationally expensive.
Why attention matters for video quality
The quality of video outputs is heavily determined by how well the model can maintain relationships across frames and across spatial regions. A model with weak temporal attention will produce inconsistent motion, flickering textures, and identity drift on characters. Strong spatiotemporal attention produces smooth motion, consistent subjects, and physically coherent dynamics.
The shift from U-Net to Diffusion Transformer architectures in video generation was partly motivated by attention scaling properties. Transformers scale attention more cleanly with model size than U-Nets, enabling better relationship modeling at larger parameter counts.
How LTX-2 implements attention
LTX-2.3 uses spatiotemporal attention as the core mechanism in its Diffusion Transformer. Every attention operation in the model attends jointly across spatial and temporal dimensions, rather than separating the two. This is a key architectural reason for the model's temporal consistency quality compared to earlier frame-by-frame or spatial-then-temporal approaches.
For developers who want to understand the full architectural detail, the LTX-2 technical report is available through the developer program page.
