- Watch the samples, not the loss. Validation clips are the real signal.
- Stop when they stop improving. Watch for underfitting and overfitting.
- Keep every checkpoint. So you can pick the best one after the run.
Previous: Setting Up A LoRA Training Run Next: Control LoRAs (IC-LoRA)
A training run is where the dataset, captions, and configuration you’ve prepared become an actual LoRA, and it commits real time and GPU compute to get there. Launching it is straightforward; the harder skill is reading what the run is doing and knowing when to stop. This chapter covers the settings worth a deliberate choice before you start, how to launch, how to tell a healthy run from one worth cutting short, and how to publish the finish ed LoRA.
A couple of settings worth checking first
Most of the config has sensible defaults, and the Configuration Reference documents the full surface. Two settings are worth a deliberate choice before you launch.
Target modules, if you’re training an identity. By default a LoRA adapts the attention layers, which is enough for a style or a simple concept. For a character or anything that has to hold a consistent identity, also target the feed-forward layers; they give the LoRA the capacity to keep the identity together, and it’s the difference between a character LoRA that holds and one that drifts.
One caveat: if you’re training audio and video together, targeting the shared blocks this broadly can bleed into the audio and degrade it. For a joint audio-video LoRA, be more selective; the Configuration Reference lists the exact video-only, audio-only, and cross-modal module patterns.
Keep your checkpoints. Set checkpoints.keep_last_n: -1 to keep every checkpoint. The default keeps only the most recent few, and you’ll often want to go back to an earlier step once you see how the run turned out. Checkpoints are cheap next to the cost of retraining.
Keep the sequence length in a workable range
Resolution and frame count together set each clip’s sequence length, the cost of training on that clip: (W/32) × (H/32) × ((F-1)/8 + 1). Treat it as a budget: aim for around 8,000 per clip, and don’t go above 15,000. Past that, previews turn to mush and the run stops learning properly. A solid default that sits in the healthy range is 1280×704×81.
This tradeoff is what drives the previous chapter’s recipes: the chain inverts frame count as resolution climbs, and a mixed run pairs a low-resolution, long-frame bucket with a high-resolution, short-frame one. If the previews fall apart or the run stops improving right after you push resolution or frames up, check the cost of your clips first, then lower the resolution, the frame count, or the rank.
Launch
Start the run with the trainer’s launch command; on multiple GPUs, wrap it with accelerate as the Training Guide shows.
Then leave it alone and watch, which is the actual skill.
Watch the run
Three things tell you how a run is going, in increasing order of usefulness.
The loss curve should trend down, but it’s noisy and jumps around, and a lower loss doesn’t reliably mean a better LoRA. Read the shape, not the wiggles, and don’t stop a run just because loss flattened.
The validation samples are the real signal, but read them the right way. The trainer makes these sample videos small and rough on purpose, so you can watch the run without waiting for a full render each time. Use them to answer “is it learning the thing?” - meaning, is the character recognizable, is the style coming through, is the movement appearing? Something that looks messy here can be clean in a real render. Judge final quality only on a full-size render, and don’t kill a run just because its previews look rough.
A tracker like Weights & Biases, if you enable it in the trainer, puts the loss and the validation videos in one place and lets you compare runs. Tag your runs (dataset version, recipe) so you can tell them apart later and easily find the ones that worked best.
When to stop
Stop when the validation samples stop getting better, not when the loss hits a specific number. Two failure patterns to watch for:
The sweet spot is the checkpoint where the target is fully there and the LoRA still follows your prompts. Because you kept every checkpoint, you can go back and pick it.
To test a finished LoRA, see whether it performs with content it wasn’t trained on. Before you train, hold back three to five clips; once you have a checkpoint you like, generate against those held-out examples. If the LoRA handles them well, you taught it the concept; if it only reproduces the clips it trained on, it memorized them, and you need more variety in the dataset.
Ship it: publish your LoRA
Once you have a checkpoint you’re happy with, publishing it makes the LoRA easy to reuse and share. The trainer can push straight to the Hugging Face Hub: set push_to_hub: true and a hub_model_id in your config, and it uploads the weights along with a model card carrying your training config, sample videos, and validation prompts.
The piece that shouldn’t be automated is prompting guidance. Your LoRA learned from long, detailed captions in one particular style, so it performs best when prompted the same way: dense and concrete, trigger-word-first if you used a trigger. Someone picking it up won’t know that unless you tell them, so put it in the README: give the trigger word, describe the caption style, and include a couple of example prompts they can copy.
That’s the core workflow: decide, build the dataset, train, monitor, and ship. One more chapter covers a different kind of LoRA: control
Next: Control LoRAs (IC-LoRA)
