Introduction

Spatial Controls

Canny

Pose

Depth

Camera Control

From Image

From Video

Motion

Sparse Tracking

Editing

Cut-on-Action

Inpainting

Audio-Visual

Who Is Talking

Audio Only

Intensity → Waveform

Controlling the generation process of video and audio models is essential for practical creative applications. However, the space of possible controls is vast: different modalities carry different types of information, and the same input (such as a mask) can have entirely different meanings depending on the context. Rather than attempting to build a single monolithic system that handles all control types, we propose AVControl, a flexible and easily extendable framework that can be rapidly adapted to new modalities, whether standard controls like depth and pose or specialized ones such as rendering Blender previews for real-time game engines (Figure 1). We build on LTX-2 [23], a joint audio-visual DiT that natively generates synchronized video and audio, making it a natural backbone for multimodal control.

The range of controls extends well beyond spatially-aligned ControlNet-style [68] modalities like depth, pose, and canny edges. We may wish to control camera motion from a single image, or re-render an existing video at a new trajectory while preserving scene dynamics. When audio is also considered, the space grows further: adapting acoustics to a text-described environment, synchronizing video with a reference audio track, and more.

Our approach draws inspiration from In-Context LoRA (IC-LoRA) [29], where a LoRA is trained on an image model to generate composite images, such as paired images side-by-side, with a learned relationship between the panels. At inference time, one half serves as the conditioning input while the other is generated via inpainting. However, for structural controls such as depth, this approach fails to faithfully follow the conditioning signal (Figure 3). We hypothesize that the large spatial distance between semantically corresponding positions in the concatenated layout weakens their interaction in the attention layers. We therefore adopted an approach inspired by Flux Kontext [3], providing the reference on a parallel “canvas,” i.e., where additional tokens in the attention layers are processed alongside the generation target. The challenge with this parallel layout is that the model must distinguish reference tokens from generation tokens. Flux Kontext [3] addresses this by introducing a new Rotary Position Embedding (RoPE) [51] dimension, which requires learning entirely new positional relationships from extensive compute and large-scale curated paired data. For video, the data cost is even more prohibitive, as temporally aligned multi-view video pairs are substantially harder to curate at scale than image pairs. Our formulation avoids both costs. LTX-2 [23] assigns a unique per-token timestep, so the model inherently distinguishes clean reference tokens from noised generation tokens (see Section 3). The only trainable component is a minimal LoRA adapter [28] on the frozen joint audio-visual backbone. Unlike methods that require new architectural components, this minimal formulation enables faithful video structural control where direct extensions of prior methods fail. Moreover, because reference and target interact through self-attention, the reference influence can be continuously modulated at inference time globally or locally – a capability unavailable to channel-concatenation methods.

Because each control modality is a lightweight, independently trained LoRA, this design enables easy extension to new controls without retraining existing ones. Unlike monolithic methods such as VACE [33], which train all controls jointly, adding a new control, whether a neural renderer for Blender meshes (Section 4.3) or a speech-to-ambient audio transformation (Section 4.3), requires only a small dataset and a short training run. The total training budget across all 13 trained modalities is ∼55K steps, less than one third of VACE’s 200K-step training run.

To accelerate inference, we further propose a small-to-large control grid that reduces the reference canvas resolution for sparse controls such as camera parameters.

To summarize, our contributions are:

• A compute- and data-efficient framework for training per-modality control LoRAs on a parallel canvas, enabling faithful video structural control and fine-grained inference-time strength modulation.
• A diverse set of independently trained control modalities, from spatially-aligned controls and camera trajectory to audio-visual applications, demonstrating the framework’s flexibility.
• A small-to-large control grid training strategy that reduces the reference canvas resolution for spatially sparse modalities, lowering latency without sacrificing control fidelity.

Related Work

Audio-Visual Foundation Models

Building on latent diffusion models [46], recent foundation models have expanded from text-to-image to text-to-video [5, 24, 66] and joint audio-visual generation [45]. A unified audio-visual backbone can share high-level semantics while learning cross-modal alignment, enabling cross-modal control: generating video from audio, audio from video, or editing one modality while preserving consistency with the other.

LoRA and Reference-Guided Generation

Low-Rank Adaptation (LoRA) [28] injects trainable low-rank matrices into frozen layers, enabling parameter-efficient fine-tuning. Its flexibility has been leveraged for diverse use-cases, including identity preservation [48], style transfer [50], motion animation [22], and multi-LoRA fusion for joint spatial–temporal video control [71].

Reference-guided generation introduces spatial inputs such as depth, pose, and masks to constrain generation beyond text. One dominant strategy is channel concatenation [6], where the conditioning signal is concatenated along the channel dimension of the noisy latents. An alternative family [3, 69] provides the reference as additional attention tokens, enabling richer interaction at the cost of a larger token budget.

Controllable video generation. Early methods adapt image control to video via ControlNet extensions [72, 70, 13] or efficient transfer [38, 56, 43]. More recent work addresses motion editing [7], in-context LoRA for pose [27], text-driven editing [40, 37], camera and object motion control [59], and sparse trajectory control [55].

Unified frameworks. UNIC [67] represents multimodal conditions as a single token sequence with task-aware RoPE. Phantom [39] and OminiControl2 [52] address subject-consistent and efficient multi-conditional generation, respectively. OmniTransfer [69] unifies spatio-temporal video transfer via task-aware RoPE biases and reference-decoupled causal learning. VACE [33] unifies diverse video tasks into a single model with shared condition units but remains limited to its training-time control set.

Camera trajectory control. ReCamMaster [1] re-renders videos at new trajectories via frame-dimension concatenation, controlling only camera extrinsics. BulletTime [58] decouples time from camera pose via 4D-RoPE, requiring 40K iterations at batch size 64. VerseCrafter [73] uses 4D geometric control via a GeoAdapter trained for 380 GPU hours. All introduce new architectural components; our camera LoRAs require only 3,000–10,000 steps and no backbone modifications.

Audio-visual control. AV-Link [25] links frozen diffusion models for cross-modal generation but lacks structural controls. EchoMotion [64] jointly models video and human motion. Audio ControlNet [75] provides fine-grained audio control without video generation. Seedance 1.5 Pro [49] is a joint audio-visual model with lip-sync but no modular control framework. For video-to-audio intensity control, ReWaS [31] and CAFA [4] train dedicated adapters on unimodal backbones using ∼160–200K samples; our framework trains a single LoRA on the joint model with ∼8K samples.

Audio-driven talking video. MultiTalk [36] generates multi-person conversational video by adding audio cross-attention layers and Label RoPE binding to a DiT backbone. Our who-is-talking modality addresses a related problem as a single LoRA on the unmodified joint audio-visual backbone, using only an abstract bounding-box activity signal.

Concurrent work. VideoCanvas [8] uses in-context conditioning for unified video completion, including inpainting, extension, and interpolation, via Temporal RoPE Interpolation on a frozen backbone. Their approach handles spatiotemporal completion but does not address structural controls such as depth and pose, camera trajectory, or audio-visual modalities. LoRA-Edit [19] uses mask-aware LoRA fine-tuning for first-frame-guided video inpainting but is limited to editing and does not support structural controls or audio. CtrlVDiff [61] trains a unified diffusion model with multiple graphics-based modalities including depth, normals, albedo, and segmentation, but uses a fixed set of controls determined at training time and does not extend to camera trajectory or audio-visual modalities.

Our approach. In contrast to monolithic models such as VACE or unified token approaches like UNIC, we train each control modality as a separate LoRA, with no new layers or input projections. Unlike Flux Kontext [3], which introduces RoPE [51] offsets, OmniTransfer [69], which uses task-aware RoPE for video, and VideoCanvas [8], which uses Temporal RoPE Interpolation, we require no positional encoding changes, relying instead on LTX-2’s per-token timestep to distinguish reference from generation tokens.

Method

An overview of AVControl is shown in Figure 2. The reference control signal is placed on a parallel canvas alongside the generation target, and a lightweight LoRA adapter is the only trainable component. We describe each design decision below.

Parallel Canvas Conditioning

A common strategy for incorporating reference signals into diffusion models is channel concatenation, where the reference is concatenated along the channel dimension of the noisy latents and fed into the diffusion model. This incurs negligible latency overhead but requires new input-projection weights.

We instead encode the reference signal through the same VAE as the generation target, producing a set of latent patch tokens. These reference tokens are concatenated along the sequence dimension with the noisy target tokens and processed jointly through the transformer’s self-attention layers. Reference tokens are assigned a clean timestep (t=0) while generation tokens carry the current noise level, allowing the model to inherently distinguish the two without positional encoding changes. Training uses the standard diffusion denoising objective, with the loss computed only on the generation tokens; reference tokens serve as clean conditioning context. A lightweight LoRA adapter on the frozen transformer is the only trainable component, applied by default to all attention projection matrices and feed-forward layers, with the exact set of target modules optimized per modality (see supplementary Table 5). While this approach increases the token count, it provides three important advantages:

1. Training efficiency. Some modalities converge in as few as a few hundred steps, with most spatially-aligned controls requiring only a few thousand. This is because we leverage the pre-trained self-attention layers for injecting the control.
2. Fine-grained reference weighting. Because reference and target interact through self-attention, we can directly scale the attention weights between target queries and reference keys. A global strength parameter uniformly scales all target-to-reference attention, providing a continuous trade-off between structural fidelity and generative freedom. Local modulation varies this scaling per token, enabling spatial or temporal fading of the reference influence (see Fig. 9 in the supplementary). Such modulation is impossible when the reference is fused at the input channel level.
3. Support for misaligned references. Channel concatenation assumes pixel-level spatial alignment. Our formulation imposes no such constraint: for example, our cut-on-action control, which re-renders a scene from a substantially different camera angle, is similar to camera trajectory control but with potentially large viewpoint changes and a different starting frame. The reference and target videos are temporally aligned but spatially different, and the parallel canvas learns this correspondence despite the lack of pixel-level alignment.

We illustrate the failure of spatial concatenation in Figure 3: a concatenation-based LoRA trained for depth-guided generation captures scene semantics but does not faithfully follow the spatial structure of the depth signal, motivating our use of a parallel canvas approach.

Per-Modality LoRA Training

Our framework is built on top of a joint audio-visual model, yet each LoRA can be trained on a single modality (either video or audio) or on joint audio-visual pairs. A video-only LoRA (e.g., depth-to-video) controls the video stream while the base model freely generates synchronized audio. An audio-only LoRA (e.g., speech-to-ambient) controls the audio stream while the base model generates accompanying video. This single-modality training keeps individual runs small and focused while the joint foundation model provides cross-modal generation at no additional training cost. We can apply a video LoRA and an audio LoRA at the same generation.

Combining Conditions

The framework conditions on a single reference signal. To combine multiple control signals, we merge them onto one canvas by compositing; for instance, masked depth overlaid with pose for neural rendering from Blender, keeping geometry aligned while allowing the model freedom on character motion.

Small-to-Large Control Grid

Not all control modalities carry the same amount of information. Dense, spatially-aligned signals such as depth maps require a relatively high-resolution reference canvas, while sparser controls like camera parameters can be expressed with far fewer tokens. We leverage this by scaling the reference canvas resolution according to the information density of each modality. This small-to-large control grid reduces the number of additional attention tokens, and consequently the inference latency and memory overhead, for modalities that do not require pixel-level reference detail; see the supplementary for details.

We next validate these design choices on standard benchmarks and demonstrate the framework across a diverse set of control modalities.

Experiments

We evaluate AVControl on standard benchmarks, demonstrate its extendability across diverse modalities, and analyze training efficiency.

Experimental Setup

We train all LoRAs on top of LTX-2 [23], a frozen joint audio-visual foundation model. Each per-modality LoRA is trained independently on a single H100 GPU; full training details are provided in the supplementary (Table 5).

We use fixed generation parameters across all evaluations: constant seed 42, CFG 1.0, and LoRA strength 1.0. The low guidance scale reflects LTX-2’s distilled inference mode; the control signal provided via the parallel canvas supplies sufficient structural context, making high guidance scales unnecessary.

For quantitative evaluation, we adopt the VACE Benchmark [33] (20 samples each for depth, pose, inpainting, and outpainting). We use the exact same published input videos and control signals as all baselines, ensuring a fair comparison against reported numbers. We report six VBench [30] metrics: Aesthetic Quality (AQ), Background Consistency (BC), Dynamic Degree (DD), Imaging Quality (IQ), Motion Smoothness (MS), and Subject Consistency (SC).

Quantitative Evaluation

Table 1 reports VBench metrics on the VACE Benchmark. Our method achieves the highest average score on all four tasks.

Depth and pose

Our method outperforms VACE by 2.9 points on depth and 2.3 on pose, while maintaining high dynamic degree (68.4 depth, 84.2 pose) and avoiding the over-constraining failure mode of methods like ControlVideo (DD of 10–25).

Inpainting and outpainting

We use the same inpainting LoRA for both tasks. Our method outperforms VACE by 3.8 points on inpainting and 2.3 points on outpainting, with gains driven by substantially higher aesthetic quality (+8.4) and imaging quality (+8.4) on inpainting.

Control Input				Ours				VACE

Figure 4 presents a qualitative comparison with VACE on the benchmark. Our outputs exhibit higher structural fidelity while maintaining natural motion and visual quality.

Extended Modalities

Beyond the benchmark controls, we demonstrate the breadth of modalities the framework supports (Figure 5). Each modality uses its own LoRA, and adding a new one requires no retraining of existing ones. We support ControlNet-style modalities (depth, pose, canny), video editing (inpainting/outpainting, detailing), and composited controls (e.g., masked depth with pose for Blender rendering). We also train a sparse tracks LoRA for point-trajectory-based motion control, similar to ATI [55], with tracks extracted via AllTracker [26]. More results are demonstrated in the supplementary video.

Camera trajectory control

We support two modes of camera control: (1) generating diverse camera motions from a single input image, and (2) re-rendering an existing video at a new camera trajectory while preserving the original scene motion. For the latter, we estimate full camera parameters (extrinsics and intrinsics, including per-frame FOV) from the source video using SpatialTrackerV2 [62], then re-render each frame at the desired camera configuration. Optionally, rendering from a different timestamp retimes the output. Training uses a synthetic dataset of synchronized moving cameras, similar to ReCamMaster [1]. Unlike ReCamMaster [1], which controls only camera extrinsics, our camera LoRAs also control intrinsics, specifically field of view (FOV). This enables simulating focal length changes and effects such as the dolly zoom (“vertigo effect”), which is impossible with extrinsics-only methods.

Table 2 compares our camera control against dedicated methods on the ReCamMaster Benchmark [1]. We evaluate on 200 randomly sampled videos across 10 trajectory types. Our method achieves the highest CLIP-F score (99.13%), surpassing ReCamMaster (98.74%). Our COLMAP-based RotErr is 6.00^∘, though SfM fails on 27% of our videos, likely underestimating the true average error. SpatialTrackerV2 tracking on all 200 videos yields 3.55^∘ (not directly comparable to baselines). While dedicated camera control architectures achieve lower rotation error, our camera LoRA is a lightweight adapter on a general-purpose audio-visual model and additionally controls camera intrinsics (FOV), which extrinsics-only methods cannot.

Audio-visual applications

Audio-visual modalities are qualitatively different from the video-only controls above: training uses audio pairs, and the generated output spans both modalities (see supplementary video for qualitative results). We demonstrate two audio modalities: audio intensity control and speech-to-ambient, plus the cross-modal who-is-talking control. Each audio LoRA is trained on audio-only pairs, yet at inference time generates both audio and video in a single joint pass: training on a single modality, deploying on both.

Audio intensity control generates audio whose temporal dynamics follow visual content, addressing the same task as ReWaS [31] and CAFA [4]. We train a single LoRA on the joint model rather than a dedicated video encoder and adapter on a unimodal backbone. The reference canvas carries the original video with its audio replaced by an energy-envelope sonification.

Table 3 compares our intensity LoRA against dedicated video-to-audio methods on 254 VGGSound [10] test samples. We report FAD, KL, IS (via PANNS [35]) and audio-visual alignment (IB via ImageBind [20]), following [4]. Our method achieves the highest IS (34.51), indicating diverse and class-distinctive audio generation. As expected, FAD and KL are higher because our model generates both audio and video jointly in a single pass rather than specializing in audio-only synthesis, and trains on 20–25× less data (7.8K vs. 160–200K samples). Notably, our IB score (0.13) is within 0.02 of ReWaS (0.15), a dedicated V2A model, despite our fundamentally different architecture: a single lightweight LoRA on a joint audio-visual backbone with no dedicated video encoder. Additionally, our audio quality (FAD) is capped by the quality of the LTX-2 vocoder.

Speech-to-ambient embeds clean speech within ambient sounds matching a text-described scene. It operates in two modes: audio replacement (AV→A), where the video’s audio is replaced while preserving the video, and joint generation (A→AV), where both audio and video are generated from the speech signal and a text prompt. The LoRA is trained on 2,600 paired samples in 5,000 steps.

Who-is-talking control demonstrates an inherently cross-modal control: given a reference encoding spatial layout (bounding boxes) and temporal activity (which speaker is active when), the model generates multi-person talking video with synchronized lip motion and audio. The reference is abstract (colored rectangles on a black background, colored when speaking and gray when silent), yet the parallel canvas maps this to realistic video with joint audio.

To compare with MultiTalk [36], we start from the same raw inputs: per-speaker identity references and separated audio tracks. We generate a first frame conditioned on the reference images, derive a bounding-box activity map via voice activity detection, and mix all tracks into a single audio signal. Our method supports an arbitrary number of speakers, unlike MultiTalk which is limited to two, and can generate audio jointly when none is provided.

Table 4 compares on the HDTF benchmark. Our method achieves strong expression quality (E-FID 0.18) and visual fidelity (FID 12.31), outperforming all baselines on both. Lip-sync scores lag behind dedicated methods, as expected for a single general-purpose LoRA without purpose-built audio cross-attention.

Ablations and Analysis

Individual LoRAs range from 200 steps to 15,000 steps, with an aggregate budget of ∼55,000 steps across all trained modalities, less than one third of VACE’s [33] 200,000 and comparable to a single specialized method like BulletTime [58] (40K iterations). See Figure 6 in the supplementary for a visual comparison.

Training convergence

Training loss plateaus early for spatially-aligned controls. For depth, VBench scores reach 81.1 at 1,000 steps and 81.6 at 3,000 steps. The rapid convergence is consistent with LoRA fine-tuning existing weights rather than training new projections from scratch.

LoRA rank

LoRA ranks 32, 64, and 128 for depth yield VBench averages of 80.9, 81.3, and 81.6, a spread of less than 1 point. We use rank 128 as the default for a small quality margin.

Parallel canvas vs. spatial concatenation

A concatenation-based baseline (analogous to IC-LoRA [29]) trained with the same data, schedule, and rank fails to follow the depth signal (Figure 3), confirming the conditioning mechanism is the critical factor.

Generalization from synthetic data

Several LoRAs (camera-from-video, cut-on-action, local edit) are trained exclusively on synthetic or generated data yet generalize to real-world videos without fine-tuning. See the supplementary for details.

Limitations

Despite these results, our framework has notable limitations. It inherits the capabilities and limitations of the underlying base model; improvements in character motion, high-frequency details, and audio quality will directly translate to better results from our method as well.

Beyond base-model limitations, our design introduces specific constraints and failure modes. Mask representation. We encode masks within the reference video rather than providing them as a separate input. This works reliably in practice, with a rare failure mode when the video contains colors similar to the designated inpainting color.

Complex character motion. When the reference depth or pose signal contains rapid, intricate character movements, the generated video may exhibit temporal jitter or implausible limb configurations.

Camera control with fast scene dynamics. Camera trajectory control from video re-renders each frame as a point cloud from the target viewpoint. In scenes with rapid, non-rigid motion, the per-frame reprojection can produce stretching or ghosting artifacts in the reference canvas, and the LoRA may faithfully reproduce these artifacts rather than correcting them.

Reference image conditioning. The most notable capability gap relative to VACE is reference image conditioning. Identity preservation is fundamentally different from the spatial and temporal controls our framework targets and is better handled by a separate, complementary mechanism.

Conclusion

We have presented AVControl, a modular framework for training controls for audio-visual foundation models. The parallel canvas, where per-token timestep disambiguates reference from generation tokens, enables faithful structural control where alternative conditioning approaches fail, without any positional encoding changes. The same mechanism allows fine-grained strength control at inference time. Combined with per-modality LoRA adapters on a frozen backbone, the framework outperforms all evaluated baselines on the VACE Benchmark while keeping the total training budget across all modalities to ∼55K steps. Adding a new control requires only a small dataset and a short training run, making the framework practical for the constantly growing space of useful controls.

The extendability of our framework is further validated by concurrent work that has already adopted it to train new control modalities: JUST-DUB-IT [9] uses our framework for video dubbing via joint audio-visual diffusion, ID-LoRA [16] extends it to identity-driven audio-video personalization, and In-Context Sync-LoRA [44] applies it to portrait video editing. These independent efforts demonstrate that the framework generalizes beyond our own set of trained modalities.

Several directions remain open: quantitative evaluation of audio-visual modalities via perceptual metrics and user studies, lightweight mechanisms for combining LoRAs at inference time, and user-specific controls trained on a handful of personal examples.

Supplementary Material

This supplementary material provides comprehensive per-modality training details (Section 7), the small-to-large control grid design (Section 8), extended ablations (Section 9), and an extended qualitative gallery (Section 10). We strongly encourage viewing the supplementary video, which demonstrates all control modalities in motion, including audio-visual results that cannot be conveyed in static figures.

Comprehensive Training Details

Each per-modality LoRA is trained independently on a single NVIDIA H100 GPU with the AdamW optimizer and a linearly decaying learning rate schedule from 1× 10-4 to 1× 10-5. We select checkpoints by generating outputs on a held-out validation set and manually ranking checkpoints by overall preference across those samples. Table 5 reports the exact configuration used for each modality.

Per-modality data construction

We describe how each modality’s training dataset is constructed.

• Depth. We estimate per-frame depth maps from real videos using Video Depth Anything [12]. The depth video is placed on the reference canvas at half resolution (2× downscale factor); the target is the original video.
• Pose. Pose skeletons are extracted from real videos using DWPose [65]. The skeleton video is placed on the reference canvas at half resolution (2× downscale factor).
• Canny / edges. Edge maps are extracted using OpenCV’s Canny detector with default thresholds (100/200). The edge video is placed on the reference canvas at half resolution (2× downscale factor).
• Inpainting / outpainting. Training pairs are constructed from real videos by applying random rectangular and free-form masks of varying size (10–60% of the frame area). Masked regions in the reference canvas are filled with a fixed green color (#66FF00). The same LoRA handles both inpainting (interior masks) and outpainting (border masks), unified by mask representation.
• Local edit. We train on the ROSE dataset [42], which provides synthetically rendered paired videos of scenes with and without target objects, including their side effects (shadows, reflections). The reference canvas contains the original (unedited) video; the target is the edited version. At inference, we typically provide an initially edited first frame produced by an image editing model, and the LoRA propagates the edit consistently across the full video without requiring an explicit mask.
• Camera trajectory (from image). We use real videos and estimate camera extrinsics and per-frame field of view (FOV) using SpatialTrackerV2 [62]. The reference canvas encodes the target camera trajectory as a canonical grid: a point cloud constructed by unprojecting the input image using a monocular depth estimate, then rendering it along the desired camera trajectory. Disoccluded regions (holes) are filled with the same green color (#66FF00) used for inpainting masks, so the model learns to inpaint them naturally. The canonical grid is placed at 4× reduced resolution alongside the input image in an image-to-video (im2vid) setup.
• Camera trajectory (from video). Training uses the SynCamVideo dataset [2], which provides synchronized multi-camera videos of 3,400 dynamic scenes rendered in Unreal Engine 5, each captured by 10 cameras. For each training pair, the video from one camera is rendered from the trajectory of another camera as a dynamic point cloud; this re-rendered video serves as the LoRA reference. Disoccluded regions are filled with the same green inpainting color (#66FF00). The target is the ground-truth video from the second camera, and the generated output is aligned to it.
• Cut-on-action. Training data is sourced from the MultiCamVideo dataset [1], a synthetic multi-camera dataset rendered in Unreal Engine 5. We use a small subset of this dataset to bootstrap initial training. Because the original dataset contains relatively small inter-camera angles, we employ a multi-angle bootstrapping strategy: after initial training, we use the LoRA itself to generate videos with progressively larger viewpoint differences, and then continue training on these generated pairs to extend coverage to substantially different viewpoints (yaw difference up to 135^∘). This iterative process yields 15,000 training samples in total.
• Video detailing (upscaling). Training pairs are constructed by downscaling real videos to create degraded inputs. The low-resolution video is upscaled with bilinear interpolation and placed on the reference canvas at full resolution; the target is the original high-resolution video. Convergence is rapid (200 steps compared to 1,000–3,000 for other modalities), consistent with the simplicity of the super-resolution mapping.
• Audio intensity. Training uses 7,800 samples from VGGSound [10]. The reference audio is a sonification of the energy envelope: the amplitude-over-time curve of the original audio is converted into an audio signal. The LoRA generates audio whose temporal dynamics follow the visual content, conditioned on this energy envelope reference. Only the audio branch is trained; the video serves as clean conditioning context.
• Speech-to-ambient. We isolate speech from real videos using Demucs (htdemucs_ft model, 5-pass) [47] and Ultimate Vocal Remover (UVR)https://github.com/Anjok07/ultimatevocalremovergui, followed by spectral gating noise reduction. The reference canvas carries the speech-only audio track; the target is the full audio including ambient sounds. Training uses 2,600 samples in 5,000 steps.
• Sparse tracks. Point tracks are extracted from real videos using AllTracker [26] and rendered as colored dots on a black canvas. The reference canvas uses a 2× downscale factor relative to the target resolution.
• Who-is-talking. Training data is generated by detecting faces with RetinaFace [17], running voice activity detection (VAD) per speaker using Silero VAD [53], and constructing an abstract reference layout: colored bounding boxes on a black background (colored when the speaker is active, gray when silent), paired with the mixed audio of all speakers. This is the only LoRA that trains all modules (video, audio, and cross-modal attention), as the task is inherently cross-modal.

Small-to-Large Control Grid

Not all control modalities carry the same information density (Figure 7). Sparser controls such as camera trajectory parameters can be expressed with far fewer tokens than dense, spatially-aligned signals, while modalities that must transfer pixel content require full detail.

By default, every video modality uses a full-resolution (1×) reference canvas. We selectively reduce the canvas resolution for modalities whose information density permits it, controlled by a reference downscale factor. Camera-from-image uses a 4× reduction, as the canonical grid encoding is inherently sparse. Dense spatially-aligned controls (depth, pose, canny) and sparse tracks use a 2× reduction; notably, this halved grid is sufficient for faithful spatial control despite the pixel-level correspondence these modalities require. All remaining video modalities (inpainting, video detailing, local edit, camera-from-video, and cut-on-action) use the full-resolution canvas, either because they must reproduce pixel-level content or because reduced-resolution variants have not yet been validated.

The reduction in reference tokens translates directly to reduced self-attention computation, since attention cost scales quadratically with the total token count (target + reference). In our measurements on an H100 GPU, a 2× canvas reduction yields a 25–35% inference speedup and a 4× reduction yields 35–50%, with the exact gain depending on the output resolution; see ablation in Section 9.

Extended Ablations

Training steps vs. quality

Expanding on the main paper’s ablation, we evaluate depth checkpoints at finer granularity: 500, 1,000, 2,000, and 3,000 steps (Figure 8). VBench average scores are 79.8, 81.1, 81.4, and 81.6 respectively, confirming that performance plateaus early. We use 3,000 steps as the default for spatially-aligned modalities, as later checkpoints yield diminishing returns (less than 0.2 points beyond 3,000 steps).

Inference-time strength modulation

We sweep the global strength parameter from 0 (no reference influence) to 1 (full reference) for depth-guided generation. At intermediate strengths, the model respects the coarse structure of the depth map while exercising more creative freedom on fine details, enabling a continuous trade-off between structural fidelity and generative diversity. The same mechanism naturally extends to temporal modulation (fading the reference influence over time) and spatial modulation (varying strength across regions), though we do not evaluate these variants here. All modulations are purely inference-time operations with no retraining required (Fig. 9).

Control grid resolution

We compare reference canvases at 1× (full resolution), 2× downscale, and 4× downscale for the camera-from-image modality. Qualitative results are comparable at all resolutions, with a 4× downscale reducing inference latency by 35–50% relative to the full-resolution canvas depending on output resolution. For dense spatially-aligned modalities (depth, pose), reducing the canvas resolution beyond 2× downscale causes some loss of structural fidelity, as expected given the pixel-level correspondence these modalities require.

Generalization from synthetic data

Several LoRAs are trained entirely on synthetic or generated data yet generalize to real-world videos without fine-tuning. Cut-on-action and camera trajectory from video train on multi-camera scenes rendered in Unreal Engine 5 (MultiCamVideo [1] and SynCamVideo [2], respectively). Local edit trains on the ROSE dataset [42], which consists of synthetically rendered scene pairs. In all cases the LoRAs transfer to diverse real-world content. Who-is-talking is a related but distinct case: its training data is generated by the base model itself rather than by an external renderer, yet the learned control still generalizes to held-out speakers and scenes. These results suggest that the parallel canvas formulation is robust to domain shift between training and inference, likely because the LoRA adapts a small number of weights while the frozen backbone retains its broad visual prior.

Extended Qualitative Gallery

We present additional qualitative results for all control modalities. Each figure shows the control input and generated output across sampled frames. All results use the default generation parameters described in Section 4.1 of the main paper. We group results by category: spatially-aligned controls, video editing, camera trajectory control, and audio-visual modalities.

Spatially-aligned controls

Figures 10–11 show results for canny edge and sparse track control.

Video editing

Figures 12–15 show video editing results: inpainting, outpainting, local edit, and video detailing (upscaling).

Camera trajectory control

Figures 16–20 show camera control results, including image-to-video, video-to-video, and cut-on-action with diverse viewpoints.

Audio-visual modalities

Figure 21 shows a who-is-talking result. Audio-only modalities (audio intensity, speech-to-ambient) are best evaluated in the supplementary video, as static frames cannot convey audio quality.