What is U-Net?
For several years, if you opened a diffusion model, you would find a U-Net inside. Stable Diffusion 1.x, 2.x, DALL-E 2, Imagen: all used U-Net as their denoising backbone. That era is ending. Understanding U-Net explains why.
Definition
U-Net is a convolutional neural network architecture originally designed for biomedical image segmentation. It was introduced by Ronneberger, Fischer, and Brox in 2015. Its defining feature is a symmetric encoder-decoder structure with skip connections between corresponding encoder and decoder layers, forming a shape reminiscent of the letter U.
In the context of diffusion models, U-Net served as the denoising network: the component that takes a noisy image and a timestep as inputs and predicts the noise to subtract.
U-Net architecture
The U-Net consists of two paths:
The encoder (contracting path) applies successive convolutional layers and downsampling operations, progressively reducing spatial resolution while increasing feature channel depth. Each stage captures features at a different scale, from fine local details to coarse global structure.
The bottleneck is the lowest-resolution, highest-channel representation where the most abstract features live.
The decoder (expanding path) applies successive upsampling operations and convolutional layers, restoring spatial resolution. At each stage, it receives a skip connection from the corresponding encoder stage, concatenating the encoder's features with the upsampled decoder features. This allows the decoder to use both global context from the bottleneck and local details from the encoder.
Skip connections are the key architectural innovation: they allow the network to preserve fine spatial information that would otherwise be lost during the downsampling steps.
Why U-Net dominated diffusion models
When diffusion models were first scaled for image generation, U-Net had several practical advantages. It was well-understood, had been extensively studied in the medical imaging community, and its multi-scale structure was a natural fit for the multi-scale denoising task in diffusion.
Attention layers were added to U-Net architectures to improve their handling of long-range dependencies, producing hybrid architectures (convolutional backbone with attention at middle and low-resolution layers) that became the standard for image generation.
Why U-Net is being replaced by Diffusion Transformers
The primary limitation of U-Net is scaling. U-Nets do not follow neural scaling laws as cleanly as transformers. Doubling the parameters of a U-Net does not reliably produce proportionally better outputs. This becomes a serious constraint when trying to scale models to the parameter counts needed for video.
Transformers, by contrast, scale more predictably. The Diffusion Transformer (DiT) architecture introduced by Peebles and Xie (2022) demonstrated that replacing U-Net with a pure transformer denoising network produced better results and scaled more cleanly with model size.
For video generation specifically, the spatiotemporal attention that transformers enable is a significant advantage. U-Net architectures were designed for 2D images and require significant modification to handle temporal relationships across video frames. Transformers handle spatial and temporal attention naturally in the same framework.
LTX-2 and the transition from U-Net to DiT
LTX-2 is built on a Diffusion Transformer, not a U-Net. The full explanation of this architectural transition and its implications for video generation quality and scaling is covered in the Diffusion Transformers explained post. The LTX-2.3 architecture and its specific design choices are documented in the technical release notes.
