How To Build A LoRA Dataset

Previous: Decide What To Train Next: Set Up the Training Run

The dataset is the most important part of training, full stop. Investing time to create a solid training dataset will result in a much better outcome. Here’s what to focus on. This chapter is about the craft, what makes a dataset good; for the exact commands at each step, keep the trainer’s Dataset Preparation guide open alongside it.

Quality over quantity

Every clip should show the target and as little extraneous content as possible. A small, clean, on-topic set beats a big messy one every time.

How many clips you need depends on how broad the concept is. A single character, a fixed likeness, or one narrow effect will need the fewest, around 25 to 50 clips. A simple motion or camera concept might need around 50 clips (control IC-LoRAs that learn from paired footage have their own sizing, covered later). A broad visual style is more conceptual and will need more clips to define the full style, often into the low hundreds. Treat these as rules of thumb, not hard requirements; there’s no magic clip count that guarantees a good LoRA.

Again, what the clips contain matters more than hitting an exact number of clips. A character needs variety, so the model learns the subject rather than one aspect of it. Setting also matters. If every clip has the character in the same kitchen (for example), the LoRA will quietly learn the kitchen as well as the character. Vary the settings so the only constant the model can latch onto is the character itself. Expect some experimentation, too. You’ll often get better results from improving or replacing weak clips than from adding more, and iterating on the dataset is a normal part of training.

Finally, also vary your shot distances and angles across your samples. If every clip is a close-up, the LoRA will be good at close-ups but struggle with wide shots. And if you’re isolating a subject on a plain background, use white rather than black.

Cut the clips

Once you’ve identified your target content, you need to cut your source video into short, scene-aware clips, commonly around five seconds each, at the longest frame count you plan to train on. There should be one continuous action in each. Avoid hard cuts in the middle of a clip. The trainer ships a splitter (split_scenes.py) that will help you do this, with options for scene length and other useful commands.

Your clips don’t all need to be the same length, but each one has to be at least as long as the frame count you train it at. During preprocessing the trainer keeps only the first F frames of each clip and skips any clip shorter than F. Since you may train the same clips at more than one length, cut each one at least as long as the longest frame count in your plan; preprocessing trims the extra frames for the shorter stages.

Frame count is only half of a clip’s size; the other half is its resolution (the width and height), and the two together are what the trainer calls a bucket. You choose the resolution at preprocessing, and you can train at more than one (see the next chapter for more on this). What matters at cut time is source quality: start from footage at least as sharp as the resolution you’ll train at, because preprocessing can shrink a clip but can’t add detail that isn’t there.

Two things happen to every clip during preprocessing that are worth keeping in mind as you select clips: they will be center-cropped to the bucket’s shape, and only the first F frames are kept. So frame your subject near the center, keep anything important away from the edges, and put the action you care about at the start of each clip.

You can also train on stills, or mix images and video, by giving images a frame count of 1. Mixing the two takes a little extra config, which the dataset-preparation docs cover.

Note: The frame count has to satisfy frames % 8 == 1, so valid values are 1, 9, 17, 25, 33, and so on in steps of 8, up through 121 and beyond; the width and height each have to be a multiple of 32. If you’re not sure where to start, about five seconds (121 frames) is a good default to use. More involved training runs can call for longer clips: some of the multi-stage recipes in the next chapter train at higher frame counts, so cut longer if that’s where you’re headed.

Write good captions

This is where most of the quality comes from, and it’s the part people tend to cut corners on. Sloppy captions teach a sloppy LoRA, so invest time up front for a better result.

Here’s what makes a good caption:

• Be dense and concrete. Describe what’s actually on screen: subject, action, setting, lighting, camera. Vague captions teach a vague LoRA.
• Stay consistent about the target. Describe constant features the same way every time, so the model anchors the concept on the same phrases. For the things that change (setting, action, lighting), describe them differently per clip. This is how the model learns what it’s free to change.
• When training audio, caption the sound too. Describe speech and sound effects, not just the picture.

The trainer offers a captioner that will create captions for your clips. Review the output before training, though, to make sure it does what you want.

Do you need a trigger word?

A trigger word is an optional handle: a distinctive token you put at the front of every caption and then include in your prompt to activate the LoRA. Use one when you want an explicit on-switch, or to keep your subject from blurring into a common word.

If you use one, pick a distinct, made-up name the base model has no strong ideas about; a stylized spelling of your subject’s name works well. Avoid common words (robot, woman, anime); LTX already has associations for those, and your LoRA will fight them. LTX reads captions with a language model, so a readable, distinctive name works well. Pass it once at preprocessing (--lora-trigger) and it’s prepended to every caption for you.

What a good caption looks like

Here’s a real caption from a released cinemagraph LoRA, an effect that keeps a scene still except for one moving element. Notice how precisely it names what moves and what stays put:

Every part of the frame is accounted for, assigned to either moving or still, so the model has nothing to guess at. That precision is what teaches the effect; it even repeats the key point (remain completely still) to drive it home.

Preprocess into latents

One step stands between your finished dataset and training. Preprocessing encodes every clip and its caption into cached latents, the compressed form the model actually trains on; from there it never touches the raw video again. The trainer’s preprocess script (process_dataset.py) handles it in a one-time pass, and the Dataset Preparation guide shows the exact command.

You run it once for each resolution you plan to train at, since the latents are baked at a specific frame count and size. Choosing those resolutions is the next chapter’s job; nothing else about the dataset changes. This is also where your trigger word gets added, so it never has to touch your captions by hand.

Before you commit to a run, it’s worth seeing what the model will actually train on. Add --decode and preprocessing writes the cropped, resized clips to .precomputed/decoded_videos (single images are saved there as PNGs), so you can open them and catch a bad crop or an over-trimmed clip in seconds.

One gotcha: if you re-run preprocessing after changing anything, the resolution, the model, or the trigger word, pass --overwrite. Without it, the script keeps the stale latents from the previous run.

With your dataset cut, captioned, and preprocessed, the next decision is what resolution to train at.

‍

Next: Set Up the Training Run