How To Fine-Tune LTX: A Practical Guide

You fine-tune the LTX model when prompting stops getting you there: a specific style, a repeatable camera behavior, a subject the base model does not know.

For most of those goals a LoRA trained on a single GPU is enough, and full fine-tuning is only worth its multi-GPU cost when you are reshaping the model's core behavior. This guide walks through both paths on LTX-2.5, from hardware and dataset prep to launching a run and generating with your trained weights.

Everything here uses the open-source LTX Trainer, which handles LoRA training, full fine-tuning, and in-context LoRA through one configuration-driven framework. Before the first command, decide which kind of training your goal actually needs, because that choice sets your hardware bill.

LoRA or full fine-tuning?

LoRA and full fine-tuning solve different problems at very different costs. A LoRA adds small adapter layers to a frozen base model, so it trains fast, fits on a single GPU, and produces a lightweight file you can attach at inference. Full fine-tuning updates every parameter in the model, which unlocks deeper change but demands a multi-GPU cluster.

When LoRA is enough

LoRA is the right default for styles, effects, and specific subjects or motions. Because it trains only adapter layers on top of a frozen base, a standard LoRA learns a look or a behavior efficiently and ships as a small file. The LTX Trainer defaults to LoRA (training_mode: "lora"), and it is where almost every project should start.

When to fully fine-tune

Full fine-tuning is for reshaping the model itself, not teaching it one trick. Setting training_mode: "full" updates all model parameters, and the documentation is explicit about the cost: it requires multiple high-end GPUs, for example 4 to 8 NVIDIA H100 80GB cards, with distributed training through FSDP. Reach for it only when adapter-based training genuinely cannot express what you need.

What is IC-LoRA?

IC-LoRA, or in-context LoRA, learns a transformation from paired videos rather than a style from single clips. It concatenates pre-encoded reference latents to the target sequence, so the model learns to turn a reference into a target: depth or pose control, colorization, deblurring, and style transfer. Use IC-LoRA when your goal is video-to-video control, not just a new appearance.

What do you need before you start?

Fine-tuning LTX-2.5 is a Linux and NVIDIA workflow with a real hardware floor, so confirm your setup before writing a config. This prerequisites check saves you from a run that fails an hour in.

Hardware and VRAM

Hardware is the gating factor. The standard LTX Trainer configuration for LTX-2.5 targets an NVIDIA GPU with 80GB or more of VRAM on Linux; the repo is tested against Python 3.12 or newer, CUDA above 12.7, and PyTorch ~=2.7. Linux is not a preference here, the trainer depends on triton, which is Linux-only.

For 32GB cards such as the RTX 5090, the trainer ships a low-VRAM config (configs/t2v_lora_low_vram.yaml) that enables INT8 quantization and other memory optimizations, so single-GPU LoRA training is reachable on high-end consumer hardware. Full fine-tuning stays firmly in multi-GPU territory.

Checkpoint and text encoder

LTX-2.5 training needs two matched pieces: the model checkpoint and the correct Gemma text encoder. LTX-2.5 uses an LTX-specific fine-tuned Gemma 4 12B encoder and does not accept Google's vanilla Gemma 4, while LTX-2.3 uses Gemma 3. The pairing is not interchangeable, the model and its encoder were trained together, so substituting either one degrades adherence.

The main LTX-2.5 release ships as a split, Comfy-aligned pack, one safetensors file per component (transformer, text encoder, VAE) rather than the single monolithic file LTX-2.3 used. Your config names the components you're loading.

If your goal is deep adaptation rather than a style or a subject, start from LTX-2.5-Pre-Trained instead of the released model. It's the raw, non-SFT base, published specifically for aggressive fine-tuning, and it bundles the 22B DiT with the video VAE, audio VAE, vocoder, and duration head alongside the matching Gemma 4 encoder.

Fine-tuning the SFT model when you meant to reshape the base is a common way to fight the model's existing preferences instead of replacing them.

Install the trainer from the repository before you begin. Installing ltx-kernels requires uv_build>=0.9.8 and a working CUDA compiler (nvcc) matching your installed PyTorch, check both first, because this is where most setups fail:

git clone https://github.com/Lightricks/LTX-2.git
cd LTX-2
uv sync --frozen
source .venv/bin/activate

Step 1: Prepare your dataset

Dataset preparation converts your raw clips into the preprocessed latents the trainer reads. You run process_dataset.py with your resolution buckets, the model path, and the text-encoder path, and it writes encoded latents and text embeddings to disk.

uv run python scripts/process_dataset.py dataset.json \

   --resolution-buckets 768x768x25 \

   --model-path /path/to/ltx-checkpoint.safetensors \

   --text-encoder-path /path/to/gemma-root

Note the frame count in the bucket. The LTX-2.5 Video VAE enforces the rule (F-1) % 8 == 0, so 25 frames is valid while an arbitrary count like 30 is not. Choose bucket frame counts that satisfy the constraint, or preprocessing will not produce usable latents. On a split LTX-2.5 pack, also pass the separate video and audio VAE paths, as the dataset preparation guide details.

Step 2: Pick a training mode

The LTX Trainer expresses every training mode through one flexible strategy, configured per modality rather than by choosing a separate class. You set is_generated: true on the modality the model should learn to generate and is_generated: false on any modality that acts as frozen conditioning, then add optional conditions like first_frame or reference.

A basic text-to-video LoRA looks like this:

training_strategy:

 name: "flexible"

 video:

   is_generated: true

   latents_dir: "latents"

 audio:

   is_generated: true

   latents_dir: "audio_latents"

The same framework covers image-to-video (add a first_frame condition), video extension (prefix or suffix), audio-to-video (freeze audio with is_generated: false), and video-to-audio Foley (freeze video). Picking a mode is picking which modality is generated and which conditions apply.

Step 3: Configure the run

Configuration is where you set the training mode and the LoRA target modules. Target modules decide which transformer layers receive adapters, and the right choice depends on whether your training touches both modalities. For any mode involving both video and audio, use short patterns so adapters land on video, audio, and cross-modal attention:

target_modules:

 - "to_k"

 - "to_q"

 - "to_v"

 - "to_out.0"

For video-only IC-LoRA, target the video attention and feed-forward layers explicitly instead, and for audio-only modes target the audio_ modules. Getting this wrong is a quiet failure: an audio-only LoRA that uses short patterns will also adapt video layers you never meant to touch. To train on a 32GB card, start from the trainer's low-VRAM config so INT8 quantization is enabled from the first step.

Step 4: Launch and monitor

Launching a run is a single command pointed at your config file. The trainer reads the YAML, loads your preprocessed latents, and starts training:

uv run python scripts/train.py config.yaml

Watch validation samples as training proceeds and stop when the LoRA reproduces your target without overfitting the dataset. You can generate audio during validation even when you are not training the audio branch, which is useful for checking that a video-only LoRA has not degraded synchronized sound. The monitoring guide covers reading these signals in detail.

Step 5: Run inference with your model

After training, you generate with your weights through the ltx-pipelines package, which loads custom LoRAs through its loras parameter. Match the pipeline to the mode you trained: standard text-to-video, image-to-video, and extension LoRAs run on the two-stage or one-stage pipelines, while IC-LoRA control runs on the IC-LoRA pipeline. Attaching a trained adapter is a matter of passing its path, as the adapter guide shows.

What goes wrong most often?

Most failed fine-tuning runs trace to a handful of avoidable mistakes. Invalid frame counts in your resolution buckets top the list, because a bucket that violates (F-1) % 8 == 0 produces latents the trainer cannot use. A mismatched text encoder is next: pairing LTX-2.5 with vanilla Gemma 4 instead of the LTX fine-tuned encoder breaks conditioning. Wrong target modules quietly train the wrong layers. And underestimating VRAM sends a standard config out of memory on a card that needed the low-VRAM path.

If your hardware cannot meet the 80GB floor and the 32GB low-VRAM path still does not fit, you do not have to abandon LTX-2.5. The hosted LTX API runs generation in the cloud with no local GPU, which is a practical on-ramp while you plan a training setup, though the API serves inference rather than training.

Summary

Fine-tuning LTX-2.5 starts with one decision: LoRA for styles, effects, and specific subjects on a single GPU, or full fine-tuning to reshape the model on a 4-to-8 GPU H100 cluster with FSDP.

The LTX Trainer runs every mode through one flexible strategy, configured by setting is_generated per modality and adding conditions. Standard training targets an 80GB NVIDIA GPU, and a low-VRAM config with INT8 quantization brings LoRA training to 32GB cards.

Prepare data with process_dataset.py using frame counts that satisfy (F-1) % 8 == 0, launch with train.py, and generate through ltx-pipelines with the loras parameter. Match the text encoder to your model version: the LTX fine-tuned Gemma 4 for LTX-2.5, Gemma 3 for LTX-2.3 — and start from LTX-2.5-Pre-Trained if you're fully fine-tuning rather than adapting.

Ready to train your first model? Start with the first-LoRA walkthrough and the new LTX Trainer.