When Should You Train A LoRA

Next In Series: Decide What to Train

Prompt LTX well and it will give you almost anything you can describe. Feed it a reference frame and it will animate what’s in it. A LoRA is for the case neither covers: when you need more control every time, across every shot. A specific face, voice, look, or move. No prompt writes its way to that, and no single frame carries it across a series. What a LoRA buys you is control and consistency: it learns the thing from your examples and reproduces it accurately, run after run.

This guide covers two kinds of LoRA. Most of it is about standard LoRAs, which learn a subject, style, or motion you activate with a prompt; the final chapter covers control LoRAs (IC-LoRAs), which drive video-to-video from a reference signal like depth or pose. Everything before it applies to both.

What’s in this guide

This guide walks the whole process of training a LoRA on LTX, in order, each chapter building on the last. The first time through, read them in sequence:

1. When to train a LoRA: what a LoRA gives you that a prompt can’t. You’re reading it.
2. Decide what to train: pin down exactly what the model should learn before you collect your data.
3. Build the dataset: preparing the clips and captions that drive how the LoRA works.
4. Set up the training run: how to train across resolutions, from a simple mixed run to the full high-fidelity chain.
5. Run and monitor training: launch the run, read the loss curve and validation samples, know when to stop, and publish the finished LoRA.
6. Control LoRAs (IC-LoRA): train depth, pose, or edge control for video-to-video.

When a LoRA is the right tool

If you can describe what you want, or hand the model a frame that shows it, and you’re fine with some variation shot to shot, just prompt or condition it. A LoRA is the right tool when you need more control: when the thing has to come out the same every time, when no prompt or input frame can supply it, or when it has to remain accurate across a whole series of shots.

These are the most common LoRA use cases:

• A specific person’s likeness. A reference photo animates one face, but push it to a new angle or a scene you have no frame for and the identity drifts. A LoRA learns the person and generates them consistently.
• A particular voice or sound. A prompt gets you “a deep male voice,” not one specific voice with its exact timbre, and an image can’t supply sound at all. A LoRA learns it from samples.
• A signature motion or mannerism. Keyframes pin the poses, but the model fills the timing between them differently every run. “Walks confidently” doesn’t lock the walk; a motion LoRA does.
• A locked camera move. “Zoom out” gives you some zoom out, different every time. A camera LoRA gives the same move, framed the same, take after take.
• A look you can’t put into words. A reference frame carries a look into one shot, and only when you have a matching frame to feed. A LoRA learns the look from examples and applies it to anything you prompt.
• A character or world that stays consistent across many shots. Conditioning holds a look within a shot, not across a series: feed a new frame each time and the character drifts. A LoRA keeps the same character, outfit, and world stable across hundreds of shots.

TL;DR

If you take nothing else from this guide, remember these three key points for LoRA training:

• Dataset first. Quality beats quantity when training. When a LoRA underperforms, the data and captions are almost always where the problem is.
• Train for what you need. Low resolution teaches motion, high resolution teaches detail. Match the recipe to the goal.
• Combine resolutions. One resolution can’t teach both motion and fine detail. Train at more than one: a single mixed run for most setups, a multi-stage chain when you have the compute for high-resolution output.

With a reason to train a LoRA, the next question is what exactly you’re teaching the model

‍

Next: Decide What to Train