How To Evaluate A Video Generation Model

Published AI model benchmarks will not tell you which video generation model to run. They were not measured on your footage, hardware, or shot types. The method that works is unglamorous: fix every variable except the model, score five axes on a prompt set you wrote yourself, and treat any number you did not produce as a hypothesis. This post covers that method rather than a ranking: after an afternoon you can rank them yourself and re-run the test when the next release lands.

Key takeaways

• Classification metrics do not transfer to generative video: there is no ground-truth frame to compare against.

• Score five axes separately: prompt adherence, temporal consistency, motion plausibility, audio sync, artifact rate.

• Fix the seed, the checkpoint, the step budget and the decoder before comparing anything.

• A frame-count constraint invalidates most naive duration tests before they start.

Why ML Metrics Do Not Transfer

ML evaluation metrics assume a correct answer exists. Accuracy, precision, recall and ROC curves compare a prediction against a label, and model performance metrics rest on that comparison. Generative video has no label. There is no correct set of frames for "a woman walks through a night market," only a distribution of plausible ones, and a clip diverging from your reference is not wrong in any measurable sense. What replaces the label is a set of axes you score independently.

What Should You Actually Measure?

Measurement on five axes, scored separately, gives a usable profile. Collapsing them into one score is where evaluation goes wrong: a model that wins three and loses two badly is not better on average if one of the two is what your work needs.

Prompt Adherence

Prompt adherence asks whether the generation contains what you specified. Score it against a checklist written before you generate: subject, action, camera, lighting, setting. Count items present rather than forming an impression, which drifts across a long session. Prompting style affects this measurement: LTX-2.5 prompting guidance calls for chronological descriptions in a single paragraph under 200 words, starting with the action. A model evaluated with prompts written for another convention underperforms for reasons unrelated to the model.

Temporal Consistency

Temporal consistency separates video models from image models, and benchmarks summarize it least usefully. Watch for objects that deform between frames, textures that crawl, and identity that drifts. Score it at two durations, because failures compound: a model that holds for one second may not hold for four. Our guide to temporal consistency in AI video covers the diagnosis.

Motion Plausibility

Motion plausibility is distinct from consistency. A clip can be perfectly stable and still show a person walking in a way no person walks. It is the hardest axis to score without a shared vocabulary, so define what a 1 and a 5 look like.

Audio-Video Synchronization

Audio-video synchronization applies only to models that generate both. LTX-2.5 generates synchronized audio and video in a single model rather than dubbing one onto the other. The two streams run through shared transformer blocks with cross-modal attention in both directions, and that shared path is what holds them in sync. Score lip-sync, foley timing and environmental acoustics separately rather than accepting audio as a bundled extra. The audio path decodes through a HiFi-GAN vocoder to a 24 kHz stereo waveform, so judge quality on that output, not a compressed export.

Artifact Rate

Artifact rate is a count, which makes it the most reliable number in the set. Generate twenty clips per model, count the ones with a disqualifying defect, and report the fraction. A 10% rate and a 40% rate are separated by a production cost no qualitative description conveys as well.

Controls That Make It Fair

Controls turn impressions into a comparison. Four matter more than the rest.

Fix the Seed

Seeds determine the initial noise, and one prompt at two seeds can differ more than two models at one seed. Fix the seed across every run, then repeat the comparison at a second seed to confirm the conclusion is not an artifact of the first. The documented LTX-2.5 quick-start passes --seed 42 explicitly.

Fix the Checkpoint

Checkpoint choice moves results more than most model-to-model differences do. LTX-2.5 publishes ltx-2.5-22b-dev-transformer-bf16.safetensors, the full model used by the guided two-stage pipelines, and ltx-2.5-22b-distilled-transformer-bf16.safetensors, which runs in far fewer steps. Comparing one model's dev checkpoint against another's fast variant produces a meaningless number. Pick a tier and hold it.

Fix the Step and Guidance Budget

Step budgets have to be equalized, and on some pipelines they are not adjustable at all. DistilledPipeline runs on 8 predefined sigmas, 8 steps in stage one and 4 in stage two, so it cannot be dialed up to match a dev-checkpoint run. When budgets cannot be matched, compare on wall-clock time or cost and say so.

Normalize for Hardware

Hardware differences swamp model differences if you let them. The LTX-2 codebase targets NVIDIA GPUs with 80GB+ VRAM, with distilled variants supported on 32GB using quantization, and recommends CUDA 13+ for optimal performance while testing against CUDA 12.7 and newer. Quantization is itself a variable: --quantization fp8-cast for bf16 checkpoints and --quantization fp8-scaled-mm for FP8 checkpoints on Hopper and newer GPUs are different numeric paths. Record the configuration with every score.

The Constraint That Breaks Duration Tests

Duration tests fail silently more often than any other part of an evaluation. The Video VAE compresses frames as F' = 1 + (F-1)/8, requiring the frame count to satisfy (F-1) % 8 == 0. Valid counts include 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, 97, and the quick start uses 121. At 25fps, 49 frames is 1.96 seconds and 57 is 2.28. A test built on "two, four, eight seconds" is not comparing what it thinks it is. Define it in frames.

Building the Prompt Set

Prompt sets should be small, fixed, and written before you see any output. Twelve to twenty prompts is enough if they stress different things: a static shot, a camera move, a human performance, a crowd, a material that reveals texture crawl such as water or fabric, a dialogue line if audio is in scope, one hard physical interaction. Reuse the identical set for every model and every re-run. Write them in the model's documented convention, chronological and specific about camera and light:

A woman in a dark green raincoat walks left to right along a wet market street at night, stepping around a puddle without breaking stride. Neon signage in red and blue reflects off the standing water. The camera tracks alongside her at chest height with a shallow depth of field, keeping her sharp while stall lights blur behind.

That prompt is scoreable. Subject, action, camera behavior, lighting and material are each present or not, which is what a prompt-adherence checklist needs.

How Do You Read a Published Benchmark?

Published benchmarks are evidence, not answers, and four questions separate the useful from the decorative. Which checkpoint tier was measured? What step budget and guidance settings? What hardware, with what quantization? How many samples per prompt, since one cherry-picked generation describes the best case, not the expected one. A benchmark that does not disclose all four is not reproducible. Open weights change this: when weights and pipelines are public, as LTX-2.5's are, you can run the vendor's configuration and check it.

Running the Evaluation

Prerequisites: a CUDA toolkit at 12.7 or newer, with 13+ recommended, an NVIDIA GPU meeting the VRAM requirements above, and the LTX-2 repository installed with uv sync --extra natten. The LTX-2.5 quick-start download set is roughly 66 GiB and includes the bundled text encoder gemma4-12b-with-proj-ltx-2.5-bf16.safetensors, required by every pipeline. Pin the video VAE decoder too: ltx-2.5-video-vae-bf16.safetensors is the diffusion decoder, ltx-2.5-video-vae-conv-bf16.safetensors the lighter convolutional one. Use one decoder throughout, then script the sweep:

uv run python -m ltx_pipelines.distilled \
  --transformer-path models/ltx-2.5/diffusion_models/ltx-2.5-22b-distilled-transformer-bf16.safetensors \
  --num-frames 121 --seed 42 --output-path output.mp4 --prompt "..."

Add the text-encoder, video-vae, audio-vae and spatial-upsampler paths as the quick start lists them.

Loop that over your prompt set with the seed fixed, change one variable per sweep, and keep the generated files. The archive is worth more in six months than the scores are today, because the next release gets tested against the same clips.

If the hardware requirements put a local evaluation out of reach, the hosted LTX API runs the same model family without a GPU, with its reference documentation at docs.ltx.video. You lose the ability to pin quantization and decoder settings, so record that limitation.

Summary

AI model benchmarks measure someone else's workload on someone else's hardware. Classification and regression metrics do not transfer to generative video because no ground-truth frame exists. A usable evaluation scores five axes independently: prompt adherence, temporal consistency, motion plausibility, audio-video synchronization and artifact rate. Four controls make it valid: a fixed seed, a fixed checkpoint tier, a matched or disclosed step budget, and a recorded hardware and quantization configuration. Define duration tests in frames: the LTX-2.5 Video VAE requires frame counts satisfying (F-1) % 8 == 0.

Write the prompt set once. Run it against every model you consider, and again when the next version ships.