# Evaling Video Slop — Maor Bril, Character.ai

## Executive summary

The challenge of evaluating AI-generated video ('video slop') is that traditional metrics like CLIP score are effective for single frames but fail to capture temporal coherence, physical plausibility, or overall narrative quality. The solution involves shifting from absolute scoring (e.g., rating 1-10) to pairwise preference comparison (is B better than A?). This process utilizes Vision Language Models (VLMs), such as Qwen3-VL, trained with Bradley-Terry loss on pairs of real and deliberately broken footage. Critically, this evaluation judge is integrated into the Continuous Integration (CI) pipeline as a regression gate, allowing developers to catch drift and errors much earlier in the generation loop when they are cheapest to fix.

## Key takeaways

- Limitations of Current Video Metrics: Tools like CLIP score are excellent for judging individual frames but cannot assess temporal incoherence, story consistency, or physical plausibility across a video's duration. The core problem is that 'video is a storytelling medium,' requiring checks on pacing and narrative flow.
- The Shift from Scoring to Comparison: Absolute scoring (e.g., rating a video's quality) is unreliable because human judgment varies greatly. The effective approach is pairwise preference, asking which of two videos tells a better story or exhibits fewer artifacts.
- Implementing the Evaluation Judge in CI: The evaluation judge (e.g., Qwen3-VL) is used as a regression gate within the CI pipeline, ensuring that every AgentX release clears an 'eval wall' calibrated against human scores before reaching users. Catching drift early significantly reduces correction costs.
- Training Data Improvement: Initial models failed because they scored the 'vibe' (gloss) rather than the substance (story/physics). The fix involved creating a dataset by pairing real, human-generated footage against AI-generated footage to prevent the model from becoming an overfitted 'AI detector.'

## Technical details

- Evaluation Metrics & Models: The system uses a combination of metrics (e.g., CLIP score, LP IPS) for frame-level analysis and employs Vision Language Models (VLMs). The model used was Qwen3-VL, chosen because it offered sufficient performance while remaining fast enough to run in the generation loop (scoring a 15-second video in ~3 seconds).
- Training Methodology: The model is trained using Bradley-Terry loss on pairs of footage, comparing A vs. B to generalize the ability to detect what is 'better' versus simply assigning a score.
- System Architecture: The evaluation process has moved from complex pipelines to an agentic workflow, allowing the system to adapt and validate its own outputs as it generates video content. The judge runs as a regression gate in CI.

## Practical implications

- Implement evaluation checks as early as possible in the generation loop to minimize correction costs.
- Prioritize relative (pairwise) comparisons over absolute scoring when building quality detectors for generative media.
- When designing CI/CD gates for AI content, ensure the judge is calibrated against human preference and measures temporal consistency, not just frame-level aesthetics.

## Topics

AI Video Generation, Generative Media Evaluation, Continuous Integration (CI), Vision Language Models (VLMs), Machine Learning Metrics, Character.ai/judgejudy, Maor Bril (Speaker)

Source: https://www.youtube.com/watch?v=b_PmGocP4rc