Training Your First LoRA on LTX

This guide walks you through training LoRAs and IC-LoRAs on LTX-2.3 — from building your dataset to running the trainer to fixing the most common problems. Treat every setting here as a starting point, not a rule: LoRA training isn't an exact science, so always test against your own goal.

Who it's for: Built for practitioners who need precise, controllable outputs — technical partners, professional teams, and community members who want full control over the model, fine-tuning and customizing it on their own data, their style, their characters, their pipeline. Sections 1–3 read without an ML background; Section 4 assumes you're comfortable on the command line.

Quick-Start Recipe

For your first run, work in this order — don't skip ahead:

• Decide what success looks like and how you'll judge it (Section 1).
• Build a small dataset — around 50 examples — and preview 10 before generating the rest.
• Hold out 3–4 clips for clean validation.
• Run a step-0 smoke test: before/after should look identical at step 0.
• Train the small set just far enough to see it moving in the right direction — then, and only then, scale up.

1. Start by Deciding What "Good" Means

Before you touch data or the trainer, write down what success looks like. This one decision drives everything downstream — including how you prepare your data.

Fidelity or motion — pick your priority

There's a real tradeoff between resolution and clip length:

• Fidelity first? Use higher-resolution examples with fewer frames.
• Motion first? Use longer clips (e.g. 121 frames), even if you have to drop resolution.

Stronger motion learning needs longer clips; stronger visual fidelity needs higher-resolution frames. You'll set resolution and frame cap during preprocessing based on this choice — so make it before anything else.

Define a measurable success test

Decide up front:

• What output quality is acceptable (sound, background, new vs. existing characters/locations)?
• How will the LoRA be used downstream (e.g. multi-keyframe vs. text-to-video)?
• How many generations per shot are acceptable?

A great objective test is a blind one: produce ~10 shots the traditional way, generate the same 10 with your LoRA (cap ~10 generations per shot, pick the best), then have ~10 reviewers judge a random mix. The bar: reviewers can't reliably tell the AI shots from the real ones.

Always test the LoRA and the inference pipeline together

You can't meaningfully evaluate one without the other — a strong LoRA on a poor inference setup will look bad, and vice versa.

‍

2. Pick Your LoRA Type

Lora - Trains to move the distribution of the generated content - good for specific style, specific character or video effects.
IC-LORA - Trains the model to get a video condition, and generate a content that is conditioned on this input‍

Lora -
Plain / character LoRA — best for capturing a character's look and motion.

IC-LoRA (in-context) — the recommended path for control and conditioning tasks: pose, depth, canny, lighting transfer, character swap. The IC-LoRA second-stage pipeline also handles audio injection reliably.

Union / stacked IC-LoRA — a single IC-LoRA can be trained to take combined controls (e.g. depth + pose together). To chain different controls (e.g. deblurring → pose), generate with one and feed the output into the next.

Know what an IC-LoRA can't do. Because it trains the control (e.g. lighting transfer) rather than fine identity details, things like exact teeth or glasses won't be reproduced unless you explicitly train on close-up data of them. Lock those details on the first-frame image instead.

‍

At a glance: matching the setup to the job

3. Build the Dataset

Dataset composition is the most common root cause of a bad LoRA. Most problems trace back to the data, not the settings — so spend your time here.

3.1 Source real assets

A good source for stock footage is Pexels — they have a free API. Write a small script that pulls assets and lets you filter by orientation, search terms, and so on, so you assemble a dataset to spec rather than by hand.

Then split everything to the right size. Source videos rarely come at the length you want to train on, so split them into clips of a fixed duration. Around 5 seconds per clip is a common default — but set it deliberately against your fidelity-vs-motion goal from Section 1: longer clips when motion matters, shorter (and higher-res) when fidelity matters.

3.2 Create before/after pairs

Paired tasks (style transfer, de-aging, relighting, and so on) need before/after pairs. If real pairs don't exist, you can generate the synthetic side however you like — any image-editing or video generation tool works. The principles matter more than the tool:

• Start from your real footage. Extract the first frame of each source clip.
• Edit that frame (e.g. the aged version, the relit version) using an image-editing model.
• Generate the paired video from the original clip plus the edited frame using a video model.
• Run generations in parallel — fire several at a time so building the dataset doesn't take all day.

3.3 When the target is synthetic, train in reverse

The target side should be real whenever possible. If you create the "after" from a real "before" using another tool, train the LoRA in the reverse direction.

Example — de-aging: start with real young footage, synthesize old versions of it, then train from synthetic-old → real-young. If you train toward the synthetic output instead, the LoRA also learns that tool's artifacts, compression issues, and mistakes. Reversing the direction makes the LoRA map from the synthetic input back to the cleaner, real target.

3.4 Engineer variance into the data

If the LoRA needs to generalize, build enough variance into the dataset so the model learns the concept, not one specific setup. For a de-aging LoRA, that means people aged 60–80 across different locations, races, and lighting conditions, with a deliberate blend of orientations — e.g. 35% landscape male, 35% landscape female, 15% vertical male, 15% vertical female.

The cautionary tale: a 243-image set of a static boat suspended in the air on a monochrome background couldn't generate the boat at sea — the model never saw it in a real environment, so it fell back on its generic prior.

Rule of thumb: include the contexts, environments, and motion you actually intend to generate.

3.5 Choose resolution and clip length for your goal

This is where your Section 1 decision pays off. Cap frames and resolution accordingly: higher resolution / fewer frames for fidelity, longer clips / lower resolution for motion. Two things we've observed:

• Restricting training to the top-resolution portion of a dataset markedly reduces smearing; mixing small and large resolutions contributes to ghosting.
• If the subject is small in the frame at generation resolution, fine details and text get lost — avoid datasets dominated by random zoom-ins when you need consistent fine detail.

3.6 Use more than one input conditioning when it helps

You can pack multiple input conditionings into a single training example — useful when the output depends on several references at once. For example, an ingredient LoRA can bake an image and a video of the ingredients plus the reference video into one example, so the model learns them jointly.

Another example — relighting: add an extra input image showing where the light comes from, so lighting is driven by that reference instead of guessed. Careful, though: the model will follow that reference closely, so a sloppy or approximate one shows up directly in the output.

Two things to get right whenever you stack inputs:

• Make each input's role unambiguous. The caption (and the layout, if you composite inputs together) should make clear what each signal is for.
• Vary the format of each input across examples — within reason. Show the same conditioning in slightly different shapes (e.g. an ingredient as 4 frames in one image and 7 in another) so the LoRA generalizes. The exception is inputs that must be precise, like the lighting reference — there you want consistency, not variance.

3.7 Caption to disentangle, not to over-describe

• Less is often more. For narrow datasets, prefer shorter captions — reference the specific subject rather than describing every detail.
• Separate the aspects you want learned. Write captions that disentangle look vs. motion vs. speech, so each can be invoked independently at inference. Good captioning helps motion, sound, character consistency, and scene understanding all at once.
• Auto-captioning is built in. The trainer supports Florence2 auto-captioning. For audio-visual sets, a common setup is a vision-language model for the visuals plus Whisper for the audio transcription.
• Use explicit tags for A/V sets, such as [SPEECH], or fold speech into the visual description (e.g. "it looks like this person is speaking or singing").
• Reinforce behavior at inference. Explicit phrasing like "the character moves their lips and says…" or "with perfect lip-sync movement as he talks" meaningfully improves lip-sync fidelity.

3.8 Preview 10 examples before committing

Before generating a full synthetic dataset, create 10 examples and review them side-by-side against your references. This catches prompt and direction problems while they're still cheap to fix.

3.9 Hold out validation videos

Leave a few videos out of training. If your dataset has 100 videos, set 3–4 aside that the model never trains on — they give you an honest validation signal during the run.
‍

‍

4. Train

On a first run you only need 4.1–4.3. The sections marked Advanced are for tuning and scaling once the basics work.

4.1 Run a smoke test first

At step 0, the before/after should look identical — that confirms the setup is wired correctly. Then train on your small dataset (~50 examples) just to confirm the model is moving toward the goal. It doesn't need to be good yet; you're looking for early signs of the right direction before committing to the full set.

4.2 Baseline settings

Reasonable starting points, to be tuned per use case:

4.3 Tune the learning rate by how the run behaves

• Steps look volatile or unstable? Lower the learning rate slightly.
• Model isn't learning, results barely improving? Raise it slightly.

The goal is a run that stays stable while still making meaningful progress.

4.4 Rank, alpha, and the scaling trap (Advanced)

Standard LoRA scales updates by alpha ÷ rank — so if you raise rank (128/256 is common for video LoRAs) without also raising alpha, the effective learning rate drops and the higher-rank adapter can perform worse. The one-line fix is rsLoRA (alpha ÷ √rank — use_rslora=True in PEFT), which the trainer doesn't enable by default.

Two caveats:

• At modest ranks (16–32) the difference is within learning-rate-tuning noise — this mainly matters at high rank.
• Trained LoRAs often use only a fraction of their nominal rank — a rank-32 LoRA compressed down to rank ~4 loses almost no quality. You can often train (or later compress) lower than you'd expect.

4.5 Mind the FPS

A LoRA overfits to the temporal spacing it was trained at; mismatching it at inference produces ghosting.

• Match training and inference FPS. Training typically runs at 24 FPS; generating far above or below that invites degradation.
• Don't exceed ~60 cond_fps. Around 60 is as high as the model should go — far beyond that is out of distribution and causes severe ghosting.
• Recommended pattern: train on the FPS you intend to use (supersample low-FPS footage by duplicating frames if needed), set processing FPS equal to the trained FPS at inference, then transcode down to your target playback FPS with ffmpeg.

4.6 Multi-GPU training (Advanced)

The trainer supports multi-GPU via DDP (one process per GPU, same batch/config on every rank). With W = number of GPUs:

• Confirm the GPUs are free — nvidia-smi should show ~0 MiB used.
• Pick the Accelerate config — configs/accelerate/ddp_${W}gpu.yaml if it exists; otherwise configs/accelerate/ddp.yaml with --num_processes $W.
• Set visible GPUs — e.g. CUDA_VISIBLE_DEVICES=$(seq -s, 0 $((W-1))).
• Launch — uv run accelerate launch --config_file <config> scripts/train.py <your config>.yaml. Confirm the log shows MULTI_GPU with W processes.
• Run in the background and tee to train.log.

Two rules that bite people: (1) do not scale the learning rate with GPU count — keep 1.5e-4 and the cosine schedule. (2) Name the output folder with the GPU count (e.g. mydataset_ic_lora_8gpu) so parallel runs stay easy to tell apart.

4.7 Watch for overtraining

• "Loud bubbling" in the audio is the canonical overtraining signal. Monitor audio quality alongside visual fidelity.
• It's usually steps, rank, or the data — not dataset size. Overtraining typically traces back to too many steps, too high a rank, or a narrow/redundant dataset with weak captions. A large dataset is fine if it's diverse and well-captioned; a big set of near-duplicates is the real problem. If you see overtraining, cut steps or rank and prune redundant data before adding more.
• Bisect with earlier checkpoints. Hold inference settings constant and test checkpoints around steps 200–500. If earlier ones look noticeably better, it's overtraining rather than a dataset problem — which is exactly why you save intermediate checkpoints.
• More steps and higher rank both increase ghosting, and they compound — back them off when you see smearing.
  ‍

5. Evaluate and Debug

When a LoRA disappoints, isolate the cause before retraining:

• Training vs. inference first. Does the artifact show up in the validation samples generated during training, or only at inference? If only at inference, the LoRA may be fine.
• Disable the LoRA. If the artifact persists with the LoRA off, it's a base-pipeline issue, not your LoRA.
• Sweep LoRA strength 0.5–1.0. A character LoRA at strength 1.0 can suppress almost all motion and ignore action prompts; lowering strength often restores motion.
• Check FPS alignment (Section 4.5) before blaming the dataset — ghosting is frequently an FPS mismatch.
• Keep clips short for IC-LoRAs. They often perform worse on longer clips (e.g. better on a 2s clip than a 4s one); chunk long generations if needed.
• When you ask for help, bring the essentials: the training YAML/config, the dataset (or representative samples), the captions, and example outputs. All four — that's what gets you actionable answers.

When in doubt, debug in this order: data → validation samples → inference setup → LoRA strength → LR/steps/rank.

6. Tools and References

• Official LTX-2 LoRA trainer — github.com/Lightricks/LTX-2. Carries the docs for settings and dataset prep; the recommended way to train.
• ComfyUI-MultiLoRALoader — github.com/phazei/ComfyUI-MultiLoRALoader. Community node for comparing LoRAs side-by-side during training.
• Data sourcing — Pexels API for free stock footage.

Settings are starting points, not guarantees — validate per use case.