Evaling Video Slop — Maor Bril, Character.ai
Summary
The challenge of evaluating AI-generated video ('video slop') is that traditional metrics like CLIP score are effective for single frames but fail to capture temporal coherence, physical plausibility, or overall narrative quality. The solution involves shifting from absolute scoring (e.g., rating 1-10) to pairwise preference comparison (is B better than A?). This process utilizes Vision Language Models (VLMs), such as Qwen3-VL, trained with Bradley-Terry loss on pairs of real and deliberately broken footage. Critically, this evaluation judge is integrated into the Continuous Integration (CI) pipeline as a regression gate, allowing developers to catch drift and errors much earlier in the generation loop when they are cheapest to fix.
Key takeaways
-
Limitations of Current Video Metrics
Tools like CLIP score are excellent for judging individual frames but cannot assess temporal incoherence, story consistency, or physical plausibility across a video's duration. The core problem is that 'video is a storytelling medium,' requiring checks on pacing and narrative flow.
-
The Shift from Scoring to Comparison
15:20
Absolute scoring (e.g., rating a video's quality) is unreliable because human judgment varies greatly. The effective approach is pairwise preference, asking which of two videos tells a better story or exhibits fewer artifacts.
-
Implementing the Evaluation Judge in CI
17:27
The evaluation judge (e.g., Qwen3-VL) is used as a regression gate within the CI pipeline, ensuring that every AgentX release clears an 'eval wall' calibrated against human scores before reaching users. Catching drift early significantly reduces correction costs.
-
Training Data Improvement
20:47
Initial models failed because they scored the 'vibe' (gloss) rather than the substance (story/physics). The fix involved creating a dataset by pairing real, human-generated footage against AI-generated footage to prevent the model from becoming an overfitted 'AI detector.'
Technical details
-
Evaluation Metrics & Models
443s
The system uses a combination of metrics (e.g., CLIP score, LP IPS) for frame-level analysis and employs Vision Language Models (VLMs). The model used was Qwen3-VL, chosen because it offered sufficient performance while remaining fast enough to run in the generation loop (scoring a 15-second video in ~3 seconds).
-
Training Methodology
920s
The model is trained using Bradley-Terry loss on pairs of footage, comparing A vs. B to generalize the ability to detect what is 'better' versus simply assigning a score.
-
System Architecture
The evaluation process has moved from complex pipelines to an agentic workflow, allowing the system to adapt and validate its own outputs as it generates video content. The judge runs as a regression gate in CI.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.