How To Decide What LoRA To Train

Previous: When To Train Next: Build the Dataset

Before you collect a single clip, you need to decide exactly what you want the LoRA to do at inference. Every later choice you make flows from this one. The questions below will help guide you through planning your dataset.

What’s the target, and what isn’t

The target is whatever has to come out the same every time. It might be a specific character or person, a look or a style, a voice or sound, or a locked camera move. That’s what the LoRA is there to hold constant.

What is not the target matters just as much: the clothing, the setting, the lighting, the props, the framing, anything you want free to vary from shot to shot at inference. You want to lock what defines the thing, and leave everything else open.

Say you’re training a specific character. The target is that character’s look, shape, and the way they move. Everything else is free: the room they’re in, what they’re doing, the time of day. You want those to change shot to shot, so the dataset should show the character across lots of different settings.

Be specific about which features recur and which vary before you cut any clips.

One shot, or many?

How widely the LoRA has to work decides how much data you need.

• Recurring (a series, a campaign, a character across many shots): it has to stay identical everywhere it appears, so the dataset has to cover the full range you’ll use it in, different angles, poses, expressions, and scenes.
• One job (a single shot, a single effect): the LoRA only has to do one thing, so the dataset can be tightly scoped and small.

This split drives dataset size, which the next chapter goes into.

Choose a training mode

You also need to decide what kind of model you’re training. This guide is about LoRAs, small adapters that are quick to train and easy to share, and that’s the right choice for almost any character, style, or effect. Full fine-tuning (retraining the whole model) is heavier in every way and needs multiple high-end GPUs; save it for when a LoRA genuinely can’t capture what you need, and lean on the trainer docs when you do.

Within LoRA, the main choice is how the model will be driven when you generate. Generating from text alone is text-to-video; starting from a still is image-to-video. You don’t have to commit to one: the flexible default for a character or style LoRA is the image-to-video setup, which uses each clip’s first frame as an occasional guide during training (a probability, not every step). The resulting LoRA works either way at inference, from a prompt or from a starting image, so it’s the safe default. Go text-only only if you’re certain you’ll never start from an image.

The trainer handles more specialized jobs too, extending a clip, filling in masked regions, adding sound to silent video, and driving generation from a reference (that last one is Control LoRAs, later in this guide). If your goal is one of those, the trainer’s training-modes docs map each to its setup.

One more piece of the mode is whether your LoRA includes audio, which is the next question.

Does it need audio?

LTX can generate audio as well as visuals, so decide up front whether the LoRA includes audio:

• Visual only: leave audio off (with_audio: false). Most character and style LoRAs don’t need it. The base model still generates audio at inference as usual; a video-only LoRA just doesn’t shape it.
• Visual plus sound: a voice, music, or dialogue tied to the picture. Turn audio on (with_audio: true), and the captions have to describe the speech and sounds, not just what’s on screen.
• Audio is the goal: a specific voice or sound signature is the thing you’re training. Audio on, and build the dataset around the sound.

Captioning for audio is covered in the next chapter.

Get clear before clipping

Once you’ve answered these questions, write the answers down. That short spec is what you’ll check each clip against as you gather your dataset.

With the target pinned down, you can collect the clips and captions that teach it.

Next: Build the dataset

‍

Next: Build the Dataset