Video-to-Video With Open Models: Restyle, Transform & Structure Control

Video-to-video AI starts with footage you already have and gives you back a modified version of that same footage. The model is not inventing motion, it is inheriting it, so your job shifts from describing a shot to deciding how much of the original to keep.

‍LTX-2.5 exposes four separate pipelines that take an existing clip as input, and picking the wrong one is the most common reason a restyle comes back looking nothing like the plan.

This post maps each pipeline to the job it was built for, then covers the constraints that bite when you run them locally.

Video-to-Video, Defined

Video-to-video generation conditions the model on an existing clip rather than on a still image or a text prompt alone. The input video constrains every frame, so motion, timing and scene geometry survive the transformation while surface appearance does not have to. That is the distinction from image-to-video, where the model has one frame of evidence and invents the rest.

Which Pipeline Does What?

LTX-2.5 ships twelve pipelines in the ltx-pipelines package. Four of them take an existing clip as input. A fifth, listed below for completeness, works from keyframe images instead. The LTX open-source documentation describes each as follows:

• ICLoraPipeline  |  Input: Video or image  |  Checkpoint it expects: Distilled  |  What it produces: Video-to-video and image-to-video transformations

• HDRICLoraPipeline  |  Input: Video  |  Checkpoint it expects: Not specified in the model listing  |  What it produces: Video-to-video with HDR IC-LoRA output: linear float via LogC3 inverse decode, for EXR export and tonemapping

• RetakePipeline  |  Input: Existing video  |  Checkpoint it expects: Not specified in the model listing  |  What it produces: A regenerated time region of that video

• DubItPipeline  |  Input: Existing video  |  Checkpoint it expects: Distilled  |  What it produces: Rephrased speech matching speaker identity and lip movements, single IC-LoRA, two stages

• KeyframeInterpolationPipeline  |  Input: Keyframe images  |  Checkpoint it expects: Not specified in the model listing  |  What it produces: Interpolation between the supplied keyframes

Two of those five expect the distilled transformer specifically, and so does DistilledPipeline. The documentation is explicit: ltx-2.5-22b-distilled-transformer-bf16.safetensors is "what DistilledPipeline, ICLoraPipeline and DubItPipeline expect." If you have internalized the rule that the dev checkpoint is the production choice, this is the exception, and downloading the wrong 22B file is an expensive mistake to discover late.

Restyling With IC-LoRA

IC-LoRA, short for in-context LoRA, is the mechanism behind LTX-2.5's video-to-video work. It trains on paired reference-target video sequences, which is what lets it learn a transformation rather than a subject. The trade-off is stated plainly in the LTX-2 documentation: in-context conditioning doubles the inference sequence length. Your memory ceiling for a video-to-video run is therefore lower than for a text-to-video run at the same resolution and duration.

Structure Control in Practice

Structure control is what separates a usable restyle from a slideshow. In LTX-2.5 it comes from the conditioning layer in ltx-core, which prepares latent states and applies image, video, keyframe, and generated-keyframe-slot conditioning. Generated keyframe slots are the interesting one: they insert extra model-generated frames at interior positions and require a checkpoint whose transformer config sets use_keyframes_abs_pos_embedding. Verify that before building a workflow around them.

For readers arriving from image models, the closest reference is ControlNet extended across time rather than applied per frame. The consequence is practical: your conditioning signal has to stay coherent frame to frame, because an inconsistent control signal produces inconsistent output no matter how good the base model is.

Keeping the Grade in HDR

HDR output is where video-to-video starts fitting a finishing pipeline. HDRICLoraPipeline returns linear float via LogC3 inverse decode, the form a colorist can work with. The standard pipelines separately accept EXR stills and EXR-frame folders through --hdr {SRGB_LINEAR,ACESCG,ACESCCT} and write half EXR frames plus a BT.2020/HLG master. A clip can therefore enter and leave the model without a lossy 8-bit round trip, which is the point of native HDR support for VFX work.

When Is Retake the Right Call?

Retake is the pipeline most teams under-use. RetakePipeline regenerates a specific time region of an existing video, so when most of a clip is correct and one gesture is wrong, you do not re-roll the whole generation and lose the frames that worked. Use ICLoraPipeline when the transformation applies to the entire clip, and RetakePipeline when the problem is localized in time.

What Do You Need to Run It?

Prerequisites first. The LTX-2 codebase is pure PyTorch and targets NVIDIA GPUs with 80GB+ of VRAM; distilled variants support 32GB with quantization. CUDA 13+ is recommended for optimal performance rather than required, and the repository is tested against CUDA 12.7 and newer. The LTX-2.5 quick-start download set is roughly 66 GiB, and every pipeline needs the bundled text encoder, gemma4-12b-with-proj-ltx-2.5-bf16.safetensors. Google's stock Gemma 4 release is not a substitute: loading checks the encoder's version against the one the checkpoint was trained with, gemma4-12b-ltx-v1.

Installation follows the documented quick start:

git clone https://github.com/Lightricks/LTX-2.git && cd LTX-2
uv sync --extra natten

The natten extra is the fastest backend for the diffusion video VAE decoder and is Linux plus CUDA only. On Windows and macOS it is skipped automatically and decoding falls back to a Triton or eager implementation, so the same command works everywhere.

You also choose a video VAE decoder. ltx-2.5-video-vae-bf16.safetensors is the diffusion decoder: better quality, longer decode, more VRAM. ltx-2.5-video-vae-conv-bf16.safetensors is the convolutional decoder: lighter, no extra dependencies. For iteration passes on a restyle, the convolutional decoder is the sane default; switch to the diffusion decoder for the render you intend to keep.

If memory is tight, the documented flags are --quantization fp8-cast with --offload cpu or --offload disk. On Hopper and newer GPUs with native FP8 support, --quantization fp8-scaled-mm applies instead and expects FP8 checkpoints rather than bf16. See the quantization formats guide for what each format costs.

None of these parameters exist on the hosted service. Every flag above belongs to the local Python pipelines in the open-source repository. The LTX API is a different surface, with a /v1/retake endpoint and the ltx-2-5-fast and ltx-2-5-pro tiers, and no quantization or decoder selection. If 80GB of VRAM is out of reach, that hosted API is the lower-barrier path.

Fixing Broken Consistency

Consistency failures in video-to-video usually trace to one of three causes, and only one of them is the model's fault.

Invalid frame counts. The Video VAE compresses frames on a fixed ratio: latent frames come out as F' = 1 + (F-1)/8, which means the input frame count must satisfy (F-1) % 8 == 0. Ask for 30 frames and the constraint is not met. Valid counts include 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, 97, and the documented quick-start example uses 121. At 25fps, 49 frames is 1.96 seconds and 57 frames is 2.28 seconds, so "two seconds" is not a value you can simply type. If you would rather not do this arithmetic, the optional duration head, ltx-2.5-duration-head-bf16.safetensors, lets you omit --num-frames and have clip length predicted from the prompt.

Prompts that fight the conditioning. LTX prompting guidance calls for detailed chronological descriptions in a single flowing paragraph under 200 words, starting directly with the action. In a video-to-video run the motion already comes from the input clip, so prompt text that redescribes camera movement competes with the conditioning instead of reinforcing it. Describe appearance, material and light. Let the footage carry the motion.

Genuine temporal drift. When neither of the above applies, the failure is in the generation itself, and the fix lives in the sampling settings rather than the pipeline choice. Our guide to temporal consistency in AI video covers the diagnosis in more depth.

Summary

Video-to-video in LTX-2.5 is four distinct pipelines, not one feature. ICLoraPipeline handles whole-clip restyling and transformation and requires the distilled 22B transformer. HDRICLoraPipeline produces linear float output suitable for EXR export and grading. RetakePipeline regenerates a bounded time region rather than the full clip. In-context conditioning doubles the inference sequence length, so memory headroom for a video-to-video run is lower than for text-to-video at the same settings. Frame counts must satisfy (F-1) % 8 == 0, and violating that constraint is the most frequent cause of results that do not match the request.

The weights are on Hugging Face and the pipelines are in the LTX-2 repository. Pull the distilled transformer, run a restyle on a clip you already know well, and compare it against the original before scaling the workflow up.