What Are IC-LoRAs & How To Use Them

Previous: How To Run LoRA Training Start From The Beginning: When to Train a LoRA

The last five chapters followed a single path end to end: building a standard LoRA, one you invoke with a prompt to bring back a character, a style, or a look. This chapter steps off that path. A control LoRA is driven at inference by a reference video: a depth map, a pose skeleton, or an edge map, rather than by a prompt, so you steer the structure and motion of a generation from an input instead of only describing it. These are called IC-LoRAs (In-Context LoRAs).

Most of the craft carries over, you still curate a dataset, caption it, preprocess, and train, but the dataset is built from paired videos and a few settings differ. This chapter covers what’s different.

A note on terms: here the reference is the control input you feed in and the target is the video you want out, a different sense of “target” than in the rest of the guide.

What an IC-LoRA is

An IC-LoRA is trained on pairs: a reference video (say, a pose skeleton) and the target video you want that reference to produce. The model learns the transformation from one to the other. At inference you hand it a new reference, and it generates a video that follows it. The same mechanism handles more than structural control: deblurring, colorization, style transfer, and restoration are all reference-to-target transformations you can train this way.

When you need one, and which signal

You don’t always need an IC-LoRA. For plain video-to-video, use the ordinary pipeline; only bring in a control LoRA when you’re actually driving generation with a control signal. When you are, the signal you pick depends on what you’re controlling:

One reality check: a control signal drives structure from the reference. It won’t invent camera moves or angles the reference doesn’t already contain.

Build the dataset

The dataset is pairs of videos, a reference and a target, with the same frame count. You generate the reference signal from your target clips with the trainer’s helper:

uv run python scripts/compute_reference.py scenes_output_dir/ \
    --output scenes_output_dir/dataset.json

It produces Canny edge references by default; edit the compute_reference() function to compute depth maps, pose skeletons, or another signal.

Two things make this affordable and reliable:

How many pairs you need scales with how complex the transformation is. A narrow, well-defined control signal can train on several hundred pairs; a demanding job needs considerably more. Lightricks’ Ingredients IC-LoRA, which conditions an entire scene on a reference sheet, was trained on roughly 2,000 pairs.

Settings

Set the strategy to video-to-video and point it at the reference latents:

model:
  training_mode: lora
training_strategy:
  name: video_to_video
  reference_latents_dir: reference_latents

On rank, keep alpha equal to rank and scale both to the difficulty of the transformation. A single, well-defined structural signal (one depth or pose stream) needs little capacity, so start low, around 16 to 32. A complex transformation needs much more: the Ingredients IC-LoRA trained at rank 128 (alpha 128). Start conservative and raise the rank if the LoRA isn’t capturing what you want.

Because an IC-LoRA processes the reference and the target together, the sequence length roughly doubles, so training and inference are slower than a plain LoRA. The reference downscale is your main lever for keeping that in check. Your validation config also needs reference_videos and a reference_downscale_factor that matches preprocessing.

Running it

Preprocess the pairs (note the reference options), train against the video-to-video config, then generate with the IC-LoRA inference pipeline.

uv run python scripts/process_dataset.py dataset.json \
    --resolution-buckets "768x448x49" \
    --model-path /path/to/ltx-2-model.safetensors \
    --text-encoder-path /path/to/gemma-model \
    --reference-column "reference_path" \
    --reference-downscale-factor 2

Train as usual, then run inference with the ICLoraPipeline from the ltx-pipelines package.

What breaks

Control LoRAs have their own failure modes. The common ones:

Control LoRAs also stack with the concept LoRAs from the rest of this guide: you can drive your character LoRA with a pose or depth reference to put your subject through a motion you control.

That’s the whole workflow, from deciding what to train through shipping a LoRA and controlling it. For the mechanics behind any step, the LTX trainer documentation is the reference.