Multimodal Conditioning In Video Generation Explained

Multimodal conditioning is the difference between a video model that guesses and a video model that follows direction. A text prompt alone gives the model semantic intent but no visual anchor.

A reference image locks composition but no motion. A reference video locks motion but no subject transformation. Real production shots need multiple signals combined, and each combination has a different way to fail.

This guide is the taxonomy of conditioning signals in LTX-2.5: when each one earns its place, how they combine, and what the pipelines actually accept as input.

What multimodal conditioning means in video generation

Multimodal conditioning means steering a video diffusion model with more than one input signal at once. LTX-2.5 is a 22B asymmetric dual-stream diffusion transformer that jointly models video and audio distributions, and its pipelines accept several conditioning signals: text prompts, reference images, reference videos, keyframes, and audio. Each one enters the denoising loop at a different place.

Every conditioning signal is steering, not scripting. Even a "locked" reference image is one input to a noisy denoising process, and the model still generates the rest. The craft is in choosing which signal locks which dimension of the output.

Why single-signal often fails on production shots

Text alone leaves too much room. "A woman walks through a rainy Tokyo street" gives semantic intent, but three generations from the same prompt produce three different women, three different Tokyos, three different rain patterns. A reference image locks identity and composition but not motion, so the model interprets "walks" freely. Combining both, a reference image for identity plus a text prompt describing the walk, gives the model both anchors, and iterations converge fast.

The four conditioning signals in LTX-2.5

LTX-2.5 pipelines accept four core conditioning types, each connected to a specific pipeline surface.

One packaging note before the details: LTX-2.5 ships as a split, ComfyUI-aligned checkpoint pack, one safetensors file per component rather than the single monolithic file LTX-2.3 used, so your config names the transformer, text encoder, and VAEs individually.

Text prompts, the baseline semantic signal

Text prompts run through a custom Gemma 4 12B text encoder that LTX fine-tuned for LTX-2.5 and ships in the model repository.

This is the single most common setup error when porting a 2.3 workflow. LTX-2.3 used Gemma 3, LTX-2.5 does not accept Google's vanilla Gemma 4, and the model and encoder were trained together, so substituting either one degrades prompt adherence rather than failing loudly.

The encoder produces separate embeddings for the video and audio streams. Video and audio contexts differ even from the same prompt, which is what lets the model synthesize speech that lip-syncs with visual mouth movements while staying natural in cadence and emotion.

LTX-2.5 also ships a dedicated prompt enhancer that expands terse input into richer cinematic and temporal instruction at near-zero additional compute. That changes the old advice.

On LTX-2.3 the guidance was to hand-write roughly 200 chronological, cinematographic words: main action first, then movements, appearances, environment, camera, lighting, changes.

On LTX-2.5 that structure is still the right shape, but you can write shorter and let the enhancer expand, which matters when you are sweeping many prompts. Write long by hand when you need deterministic control over a specific shot. Let the enhancer work when you are exploring.

Reference images, composition and subject identity

Reference images enter through ImageConditioningInput (from ltx_pipelines.utils.args) and attach to a specific frame with a strength value. There are two image conditioning methods, and picking the right one matters.

Replacing latents. Used by TI2VidOneStagePipeline, TI2VidTwoStagesPipeline, TI2VidTwoStagesHQPipeline, DistilledPipeline, and ICLoraPipeline. The pipeline encodes the image and replaces the latent at the target frame. This is strong control, and the reference frame will be very close to the image. Use it when you have a specific opening frame or subject you must preserve.

Guiding latents. Used by KeyframeInterpolationPipeline. The image is added as a guiding signal rather than replacing the latent. This is softer control, and the model transitions smoothly through the frame rather than snapping to it. Use it when you have multiple keyframes and want a smooth path between them.

Reference videos, motion and structural transfer

Reference video conditioning runs through ICLoraPipeline. The pipeline takes a reference clip that is positionally aligned to the generated video and conditions on it via IC-LoRA, an in-context LoRA trained to interpret paired reference-target sequences.

The reference clip can be lower resolution than the target, which costs fewer tokens.

Control modes cover pose skeletons, depth and edge maps, motion tracks, relighting, and detail refinement. ComfyUI ships an LTX-2.5 Union-Control workflow against the distilled checkpoint.

Most IC-LoRAs published for LTX-2.3 also work on LTX-2.5, so the 2.3-named adapters on HuggingFace are a valid starting point: Union-Control combines multiple control signals in one adapter, Motion-Track-Control transfers motion from the reference, Relight changes scene lighting, Pose-Control conditions on pose signals, and Detailer refines detail.

Test each adapter against your checkpoint before committing a workflow to it, since cross-version compatibility is the norm rather than a guarantee. As with LTX-2.3, ICLoraPipeline only works with a distilled model checkpoint.

Audio, the dual-stream signal

Audio conditioning enters through A2VidPipelineTwoStage. The input audio is encoded via the Audio VAE into audio latents and used as the initial state for the audio stream during denoising. Stage 1 generates video at half resolution with the audio frozen. Stage 2 upsamples 2x and refines the video while keeping the audio fixed, using a distilled LoRA.

The original audio waveform is passed through and returned in the output MP4 to preserve fidelity. On the generation side, audio latents decode to a 24 kHz stereo waveform through a HiFi-GAN-based vocoder.

For dialogue specifically, LTX-2.5 also exposes LipDubPipeline, which is the better surface when the job is fitting speech to existing footage rather than generating a shot and its audio together.

How does LTX-2.5 combine signals?

LTX-2.5 combines conditioning signals through its dual-stream architecture and the MultiModalGuiderParams configuration. During each denoising step, three guidance signals are computed and combined.

CFG (Classifier-Free Guidance). Steers generation toward the text prompt by computing (cond - uncond_text). Controlled by cfg_scale. Typical values 2.0 to 5.0. Set to 1.0 to disable, which is where distilled checkpoints run.

STG (Spatio-Temporal Guidance). Improves structural coherence by perturbing specific transformer blocks and steering away from the perturbed prediction. Controlled by stg_scale and stg_blocks. Typical values 0.5 to 1.5. Set stg_scale=0.0 or stg_blocks=[] to disable.

Modality CFG. For joint audio-video generation, steers the model away from unsynced audio and video results. Controlled by modality_scale. 3.0 is the working default for audio-video jobs, and 1.0 disables it for video-only generation.

The values in the configurations below are the documented defaults carried forward from LTX-2.3. They are a working starting point on LTX-2.5, but scale values and block indices are checkpoint-specific and 2.5 changed both the text encoder and the decode path. Confirm them against the example configs shipped with your checkpoint and packages/ltx-pipelines/docs/conditioning.md before you build a recipe on them.

Combining text and reference image, the production default

The most common production pattern is text plus reference image on TI2VidTwoStagesPipeline. The reference image locks composition and identity at frame 0. The text prompt drives motion and camera. A concrete configuration for a video-only job, with audio guidance disabled:

python

video_guider = MultiModalGuiderParams(
   cfg_scale=3.0,       # moderate prompt adherence
   stg_scale=1.0,       # temporal coherence on
   rescale_scale=0.7,   # variance rescale
   modality_scale=1.0,  # video-only, sync disabled
   stg_blocks=[29],     # perturb the last block
)

Combining text, image, and audio for synchronized shots

When audio matters, whether for dialogue, music-driven cuts, or foley-locked action, the pipeline is A2VidPipelineTwoStage with both a reference image and a text prompt. Modality guidance turns on:

python

audio_guider = MultiModalGuiderParams(
   cfg_scale=7.0,       # stronger audio prompt adherence
   stg_scale=1.0,
   rescale_scale=0.7,
   modality_scale=3.0,  # video-audio sync on
   stg_blocks=[29],
)

The higher cfg_scale on the audio guider, 7.0 against 3.0 for video, is a documented default in the pipeline docs. Audio benefits from stronger prompt adherence. And modality_scale=3.0 is the value that actually steers away from unsynced generations.

Combining reference video and text for video-to-video restyling

ICLoraPipeline conditions on both a reference video and a text prompt. Pick the IC-LoRA that matches the control mode you need, load it into the distilled model, and pass the reference clip plus a target prompt. Extract the control signal, whether a pose skeleton, depth map, or motion track, before you call the pipeline. The pipelines do not ship a depth estimator.

When signals conflict, the weighting problem

Every conditioning signal is a bias on the denoising step. When two signals want different things, the model averages them, and the result is often neither.

Strong reference image plus a prompt describing a different subject. The image says "this face" and the prompt says "a different face." The model produces a blend that looks like neither. Reduce image conditioning strength below 1.0 to let the prompt reshape the subject, or match the prompt to the image and describe the target as a transformation.

High CFG plus high STG. Both push the output away from the unconditional prediction. Together they over-saturate and produce artifacts. Keep one high, typically CFG for prompt adherence, and the other moderate around 1.0. Use rescale_scale=0.7 to bring variance back in line.

Audio prompt describing action B while the video prompt describes action A. Modality CFG will try to sync them and produce ambiguity. Audio and video prompts should describe the same scene from different angles, visual against auditory, not different scenes.

Common failure modes and fixes

Over-conditioning produces rigid, artifact-ridden output. A strong reference image, a high-weight IC-LoRA, and a maxed-out CFG at once leave the model no freedom to denoise cleanly. Symptoms are static-looking motion, texture artifacts, and unnatural transitions. Reduce conditioning strength on one signal, usually the image, down to 0.7 or 0.8, and lower CFG toward 3.0.

Under-conditioning produces drift. Text alone at low CFG on a complex scene gives the model too much room. Subject identity drifts across frames and composition wanders. Add a reference image at frame 0 to anchor, or raise cfg_scale toward 5.0.

Frame count violates the VAE constraint. The Video VAE requires frame count to satisfy (F-1) % 8 == 0. Requesting an intuitive count like 30 or 100 frames silently produces unexpected behavior. Pick a valid count: 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, 97, 121. At 25fps, 121 frames is 4.8 seconds. LTX-2.5's Auto Duration can predict clip length from scene content instead, which removes the manual arithmetic when you do not need a fixed length.

Wrong encoder carried over from a 2.3 workflow. Prompt adherence degrades without an obvious error. Confirm you are loading the LTX-2.5 Gemma 4 encoder, not Gemma 3 and not stock Gemma 4.

Retake source has wrong dimensions. RetakePipeline requires the source video's frame count to satisfy the 8k+1 format and its resolution to be a multiple of 32. A raw camera clip usually needs a preprocessing pass before Retake will accept it.

Can you use depth maps or pose signals directly?

Depth and pose conditioning are exposed through the pre-trained IC-LoRAs: Pose-Control for pose skeletons, and Union-Control for combined signals that include depth and edge modes. The workflow is to extract the depth or pose signal from a reference video using external tooling, load the matching IC-LoRA into a distilled checkpoint via ICLoraPipeline, and pass the reference plus a text prompt. Standalone depth maps as a separate input tensor are not exposed on the pipeline surface. The IC-LoRA is the path.

The hosted API is a shorter path when local pipelines aren't practical

Every conditioning signal above is available on the open-source Python pipelines, and running them locally has a real hardware floor.

The standard LTX-2.5 configuration targets an NVIDIA GPU with 80GB or more of VRAM, with FP8 and NVFP4 quantization plus block streaming bringing generation onto 32GB cards.

If your hardware does not clear the bar, the hosted LTX API runs the same generation stack with a different parameter surface. cfg_scale, stg_scale, and rescale_scale are open-source Python knobs and are not exposed by the API, so a recipe developed on the local pipelines transfers only with explicit mapping. Write that mapping down before you build a workflow that depends on it.

Summary

Multimodal conditioning in LTX-2.5 combines text prompts (via the custom Gemma 4 12B encoder, not Gemma 3 and not stock Gemma 4), reference images (via ImageConditioningInput, using either replacing or guiding latents), reference videos (via IC-LoRA on ICLoraPipeline, distilled checkpoints only), and audio (via A2VidPipelineTwoStage, or LipDubPipeline for dialogue on existing footage).

Signals combine through MultiModalGuiderParams, which controls CFG for text adherence, STG for temporal coherence, and modality CFG for audio-video sync.

Most LTX-2.3 IC-LoRAs carry over, but test before you commit. Every pipeline respects the frame-count constraint (F-1) % 8 == 0, or you can let Auto Duration pick the length. The two failure modes to watch are over-conditioning, which produces rigid output, and under-conditioning, which produces drift.

Pick the signals that lock the dimensions you care about, and let the model handle the rest.

Start layering signals

The LTX-2 repository ships every pipeline referenced here. Pull the split checkpoint pack and the IC-LoRAs from HuggingFace, or test text plus reference image without a local install in the LTX Playground.