How To Train A Character LoRA With LTX-2.5

A character LoRA for LTX-2.5 is a standard LoRA trained on clips and stills of one character, with a trigger token that calls that character back in any prompt. Two decisions made before the first step decide whether it works: which training mode you pick (text-to-video or image-to-video) and whether the adapter learns the character's voice. This guide walks through both, plus the dataset, config, and checkpoint choice, using the LTX Trainer.

Prerequisites. The trainer runs on Linux with CUDA because it depends on triton, which is Linux-only; the repo tests CUDA 12.7 or newer and recommends 13 or newer. The standard config expects an 80GB GPU, while t2v_lora_low_vram.yaml targets 32GB cards such as the RTX 5090 using INT8 quantization and other memory optimizations. LTX-2.5 also needs its LTX-specific Gemma 4 text encoder; Google's stock Gemma 4 is not a substitute. If that hardware is out of reach, the LTX API covers generation without local GPUs, though it does not train LoRAs.

Does your character need a LoRA?

Character consistency in LTX-2.5 has a cheaper first option, and the choice is really between two tools:

• Image-to-video. Conditions the first frame on an image of the character and animates from there. Free of any training, but it holds for one shot at a time and needs a matching first frame for every shot.
• A character LoRA. Trains the character into an adapter you can call from any prompt, in any framing, with or without a reference frame. Worth the training time once you need the same character across many prompts and scenes.

The LoRA earns its training time exactly where image-to-video breaks: many prompts, many framings, no matching first frame for each.

Build a character LoRA dataset

Character LoRA datasets are metadata files (JSON, JSONL, or CSV) pairing each clip with a caption. The trainer ships scripts for each stage:

1. Split long footage. split_scenes.py input.mp4 scenes/ --filter-shorter-than 5s cuts raw video into coherent scenes.
2. Caption. caption_videos.py scenes/ --output dataset.json writes one detailed paragraph per clip covering visuals and audio, using a local Qwen3-Omni server or Gemini.
3. Review every caption. The docs warn auto-captions can contain hallucinated content.
4. Preprocess with a trigger. process_dataset.py with --lora-trigger "MYCHAR" prepends the token to every caption, so it never goes into the JSON by hand.
5. Decode to check. Add --decode to write the cropped, encoded clips back out and confirm the character is still in frame.

Step 5 matters for characters specifically. Preprocessing resizes each clip, center-crops the longer side, and keeps only the first frames the bucket allows. A wide two-shot can lose your character to the crop, and the LoRA then learns the wrong person.

Stills are allowed. Turnaround sheets and expression studies train with a frame count of 1 in the bucket. Mixing stills and clips needs multiple buckets, such as "768x448x1;768x448x89", and batch_size: 1. For high-quality, detailed video, the docs recommend larger spatial dimensions with fewer frames (for example 768x448 at 89 frames), a good fit when facial detail matters more than long motion. The dataset guide covers this in depth.

T2V or I2V training mode?

T2V and I2V are both documented modes of the flexible training strategy. T2V denoises video and audio from text only. I2V adds a first_frame condition: the first latent frame is kept clean and excluded from loss, and the probability value sets how often that happens, with the remaining samples training as pure T2V.

For a character LoRA, I2V mode with probability: 0.5 is the practical default. Half the steps teach the model to produce the character from text alone, and half teach it to continue from a frame of the character without drifting. You get an adapter that serves both entry points: generation from a prompt, and continuation from a reference frame. Pick pure T2V only if you will never start from a reference image.

Should the LoRA learn the voice?

Voice follows from target_modules. LTX-2.5 has separate video and audio branches, plus cross-attention between them. The docs recommend short patterns (to_k, to_q, to_v, to_out.0) for audio-video training, because they match video, audio, and the cross-modal modules that keep speech in sync with lip movement.

• Character with a consistent recorded voice: keep audio generated and use the short patterns.
• Silent character, or clips whose audio is music and crowd noise: target only the video modules (attn1.* and attn2.*), so the adapter does not absorb audio you never want back.

Mixed datasets are the trap here. If half the clips carry the character's lines and half carry a soundtrack, a full audio-video LoRA may learn a blend of both.

A starting character config

This config assembles documented starting values into one file for LTX-2.5's split checkpoint layout. It trains on the dev transformer, which is the trainable checkpoint, so the resulting adapter pairs with the guided two-stage pipelines rather than the distilled path. The values come from the configuration reference; the combination is a suggested starting point, not an official preset.

yaml

model:
 model_path: "/models/ltx-2.5/diffusion_models/ltx-2.5-22b-dev-transformer-bf16.safetensors"
 text_encoder_path: "/models/ltx-2.5/text_encoders/gemma4-12b-with-proj-ltx-2.5-bf16.safetensors"
 video_vae_path: "/models/ltx-2.5/vae/ltx-2.5-video-vae-bf16.safetensors"
 audio_vae_path: "/models/ltx-2.5/vae/ltx-2.5-audio-vae-bf16.safetensors"
training_mode: "lora"
lora:
 rank: 32
 alpha: 32
 target_modules: ["to_k", "to_q", "to_v", "to_out.0"]
training_strategy:
 name: "flexible"
 video:
   is_generated: true
   latents_dir: "latents"
   conditions:
     - type: first_frame
       probability: 0.5
 audio:
   is_generated: true
   latents_dir: "audio_latents"
optimization:
 learning_rate: 1e-4
 steps: 2000
 batch_size: 1
 enable_gradient_checkpointing: true
checkpoints:
 interval: 250
 keep_last_n: -1

Every bucket needs width and height divisible by 32 and a frame count that satisfies (F-1) % 8 == 0, so 49, 89, and 121 work while 48 or 100 fail. The training modes docs list every other mode. Start with uv run python scripts/train.py and your config path.

How do you pick the checkpoint?

Checkpoint choice is where character LoRAs are won. With keep_last_n: -1, every 250-step checkpoint stays on disk as lora_weights_step_XXXXX.safetensors, so you can step back when later ones overfit.

Validate with the character's own first frame. The troubleshooting guide recommends first-frame conditioned validation over pure text-to-video because it is more reliable. Add two samples to the validation block: one prompt with the trigger and a first-frame image of the character, and one text-only prompt with the trigger in a new setting. The second sample tells you whether identity survives without a reference.

• Identity is weak: the docs suggest raising rank to 64 or 128, training to 3,000 steps, and re-checking captions.
• Identity is strong but every scene looks the same: overfitting. Step back to an earlier checkpoint; the docs note lower learning rates often produce better results.
• Out of memory: use the low-VRAM config, load_text_encoder_in_8bit: true, or optimizer_type: "adamw8bit".

For monitoring a run end to end, see how to run and monitor LoRA training.

Run the character LoRA

Because the adapter is trained on the dev transformer, it runs in TI2VidOneStagePipeline or TI2VidTwoStagesPipeline, which accept custom LoRAs through the loras parameter. In ComfyUI, the LoRA guide calls for the LoraLoaderModelOnly node, wired to both the Stage 1 and Stage 2 samplers in two-stage templates. Put the trigger token in every prompt, keep the seed fixed while tuning strength, and test at 1.0 first. To publish the adapter, set hub.push_to_hub: true and the trainer uploads weights, a model card, and sample GIFs. The LoRA adapters guide covers loading in full.

Summary

A character LoRA for LTX-2.5 is a standard LoRA trained with the LTX Trainer on clips and stills of one character, called back by a trigger token set with --lora-trigger. I2V mode with first_frame probability 0.5 produces an adapter that works from text and from a reference frame. Short target_modules patterns teach the character's voice; video-only modules keep dataset audio out. Keeping every checkpoint and validating with a first-frame image lets you pick the step before overfitting. Frame counts must satisfy (F-1) % 8 == 0, and the standard config expects an 80GB GPU, with a 32GB low-VRAM option.

Gather clips of your character, run --decode on the preprocessed set, and start your first run from the config above.