What Training Data Do You Need To Fine-Tune A Video Generation Model?

To fine-tune a video generation model, you need short video clips or images paired with captions and preprocessed into resolution buckets the model's VAE accepts. With the default LTX VAE, width and height are multiples of 32 and frame counts satisfy frames % 8 == 1, though the trainer derives the actual compression factors from the checkpoint. Get the resolution rules and your captions right, and most fine-tuning problems disappear before training starts.

This guide covers what the data actually has to look like, how much of it you need for different training approaches, and the mistakes that quietly break a run. The current LTX Trainer supports LTX-2, LTX-2.3, and LTX-2.5 through the same configuration framework.

Prerequisites: The LTX Trainer requires a supported LTX model checkpoint, its matching Gemma text encoder, Linux with CUDA, and a GPU with enough VRAM. For LTX-2.5, use the trainable dev transformer with the LTX-specific fine-tuned Gemma 4 encoder, not Google's vanilla Gemma 4, which will fail the checkpoint compatibility check. Older LTX-2 and LTX-2.3 checkpoints use their matching Gemma 3 encoder.

The standard configuration recommends 80GB VRAM, while the low-VRAM configuration brings LoRA training to 32GB GPUs such as the RTX 5090 using INT8 quantization and other memory optimizations.

What does training data do?

Training data teaches the model a behavior it does not already have: a visual style, a subject, an effect, a transformation, or a domain-specific behavior. The current LTX Trainer uses a unified conditioning framework that supports 13+ training modes, including text-to-video, image-to-video, video and audio extension, inpainting and outpainting, audio-to-video, video-to-audio, and IC-LoRA transformations.

Standard LoRA adds lightweight trainable adapters while the base model stays frozen, while full fine-tuning updates all model parameters for larger adaptations. IC-LoRA learns transformations from paired reference and target data.

LoRA, full fine-tune, or IC-LoRA?

LoRA is the right default for most focused projects. Standard LoRA training adds lightweight, portable adapters while keeping the base model frozen and can run on a single GPU. Choose full fine-tuning when you're making a larger model adaptation and have multiple high-end GPUs available; the current LTX Trainer documents distributed FSDP training on setups such as 4 to 8 H100 80GB GPUs.

Choose IC-LoRA when your goal is a transformation such as depth or pose control, style transfer, deblurring, or colorization, because IC-LoRA learns from paired reference and target data. The data you gather follows directly from that choice: standard examples for LoRA or full fine-tuning, and paired data for IC-LoRA.

How much data do you need?

How much data depends on the training approach, and the LTX Trainer does not prescribe a fixed clip count. LoRA requires less memory and compute than full fine-tuning, which makes it better suited to focused goals such as a specific style or effect. Full fine-tuning updates the entire model and requires substantially more compute.

The practical takeaway: start with the smallest dataset that cleanly represents the behavior you want, confirm it works, then expand. A tightly scoped dataset of consistent, on-target examples is generally more useful than a large, noisy one.

What makes a good training clip?

A good training clip is short, coherent, on-target, and sized to a valid resolution bucket. The trainer organizes videos into buckets defined as width by height by frames, and each clip is assigned to the nearest matching bucket. Datasets can contain videos, images, or a mix of both. For images, use a frame count of 1 in the bucket, for example 960x544x1.

Mixed image-and-video datasets require multiple resolution buckets covering both frame counts and a training batch size of 1.

Resolution and frame rules

Resolution and frame counts are constrained by the checkpoint's Video VAE. With the default VAE, spatial dimensions must be multiples of 32 and frame counts must satisfy frames % 8 == 1, which yields values such as 1, 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, 97, and 121. The preprocessing scripts derive the actual spatial and temporal compression factors from the checkpoint, so these constraints can differ with a different VAE.

The documented guideline is to use larger spatial dimensions with fewer frames for detailed clips (for example 768x448 at 89 frames) and smaller spatial dimensions with more frames for motion-heavy clips (for example 512x512 at 121 frames). Memory scales with both dimensions, and the transformer sequence length is (H/32) * (W/32) * ((F-1)/8 + 1), so a 768x448x89 clip works out to 4,032 tokens.

During preprocessing each video is resized to the bucket, center cropped on the larger dimension, and truncated to the first X frames.

Captioning your clips

If your clips lack captions, the trainer can generate them with caption_videos.py, using either the local qwen_omni captioner or the gemini_flash API captioner. Generated captions can describe both visual content and audio, including speech, music, and ambient sounds.

The documentation is explicit that automatic captions can contain inaccuracies or hallucinated content, so review and correct your dataset.json before preprocessing. You can also set a trigger word with --lora-trigger, which is prepended to every caption automatically and later activates the LoRA when you include the same token in a prompt.

Preparing the dataset

Preparing the dataset follows a three-step workflow. First, optionally split long footage into coherent scenes:

uv run python scripts/split_scenes.py input.mp4 scenes_output_dir/ --filter-shorter-than 5s

Second, optionally caption the clips into a dataset.json.

Third, preprocess to compute and cache latents and text embeddings:

uv run python scripts/process_dataset.py dataset.json --resolution-buckets "960x544x49" --model-path /path/to/ltx-2-model.safetensors --text-encoder-path /path/to/gemma-model

Audio latents are extracted from video automatically unless you pass --skip-audio. For IC-LoRA, add a reference_video column to the dataset; process_dataset.py detects it automatically. The metadata file can be CSV, JSON, or JSONL. The current convention uses caption and video columns, with media_path still accepted as a legacy alias for video.

When you preprocess with multiple resolution buckets separated by ;, set the training batch size to 1. The output lands in a .precomputed directory containing latents, conditions, and any mode-specific audio, reference, or mask data.

If you're switching an existing dataset from LTX-2.3 to LTX-2.5, reprocess the text features with the matching Gemma 4 encoder. Gemma 3 and Gemma 4 condition embeddings are not interchangeable, so use a fresh .precomputed output directory or pass --overwrite.

What are the common data mistakes?

Data mistakes, not model settings, cause many failed fine-tunes. The first is an invalid resolution bucket: with the default VAE, spatial dimensions must be multiples of 32 and frame counts must satisfy frames % 8 == 1, though the actual factors are derived from the checkpoint. The second is mixing images and videos without configuring separate frame-count buckets and setting the training batch size to 1.

The third is trusting raw auto-captions, which can contain inaccuracies or hallucinated content and should be reviewed before preprocessing. For IC-LoRA, each target needs a corresponding reference covering the same content. Reference videos can be lower spatial resolution or temporally subsampled, as long as the chosen reference scale factors are handled consistently during preprocessing and validation. Lightricks publishes a Canny Control example dataset showing the paired structure.

Summary

Fine-tuning LTX starts with captioned video clips, images, or both, preprocessed into resolution buckets compatible with the model's VAE. With the default VAE, spatial dimensions are multiples of 32 and frame counts satisfy frames % 8 == 1, though the Trainer derives the actual factors from the checkpoint. LoRA is the lighter-weight option for focused adaptations, while full fine-tuning updates the entire model and requires multiple high-end GPUs.

IC-LoRA uses paired reference and target data for transformations such as depth or pose control, style transfer, deblurring, and colorization. For LTX-2.5, use the matching LTX-specific Gemma 4 encoder and regenerate cached text features when moving from an older Gemma 3-based checkpoint. Review automatically generated captions before preprocessing, keep the dataset focused, and expand only after a clean training run.