The Video Signal

Topic

Vision Language Models (VLMs)

All digests tagged Vision Language Models (VLMs)

From Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam thumbnail

· 21:09

From Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam

Sarvam introduced a 3 billion parameter (3B) State-Space Vision Language Model (VLM) designed for document AI, achieving state-of-the-art (SOTA) performance while being small enough to run on a single GPU. The model is notable for its use of a State Space Model (SSM) backbone instead of a traditional Transformer, which efficiently handles the quadratic complexity of long documents. The training process is a sophisticated four-stage curriculum, covering 13 trillion tokens across English and 22 Indian languages, and includes a proprietary data engine and machine-checkable reinforcement learning (RL) for scalable OCR correctness.

Key takeaways

  1. SOTA Document AI Performance

    The 3B parameter model achieves SOTA in document AI, claiming to beat models 100 times larger in size.

  2. Architectural Innovation (SSM) 3:30

    The model uses an SSM backbone instead of a Transformer to manage the high compute costs associated with quadratic attention ($L^2$) when processing long documents (up to 10,000 visual tokens).

  3. Comprehensive Training Curriculum 6:00

    Training involves four stages: 13T token pre-training (text), 300M image-text pairs (continual pre-training), 100M OCR samples (SFT), and RL using machine-checkable unit tests for rewards.

  4. Sovereign Capability 2:30

    The entire model building, data, compute, and training were done end-to-end in India, addressing the lack of machine-readable data for Indian languages.

Watch on YouTube Full article

Evaling Video Slop — Maor Bril, Character.ai thumbnail

· 23:13

Evaling Video Slop — Maor Bril, Character.ai

The challenge of evaluating AI-generated video ('video slop') is that traditional metrics like CLIP score are effective for single frames but fail to capture temporal coherence, physical plausibility, or overall narrative quality. The solution involves shifting from absolute scoring (e.g., rating 1-10) to pairwise preference comparison (is B better than A?). This process utilizes Vision Language Models (VLMs), such as Qwen3-VL, trained with Bradley-Terry loss on pairs of real and deliberately broken footage. Critically, this evaluation judge is integrated into the Continuous Integration (CI) pipeline as a regression gate, allowing developers to catch drift and errors much earlier in the generation loop when they are cheapest to fix.

Key takeaways

  1. Limitations of Current Video Metrics

    Tools like CLIP score are excellent for judging individual frames but cannot assess temporal incoherence, story consistency, or physical plausibility across a video's duration. The core problem is that 'video is a storytelling medium,' requiring checks on pacing and narrative flow.

  2. The Shift from Scoring to Comparison 15:20

    Absolute scoring (e.g., rating a video's quality) is unreliable because human judgment varies greatly. The effective approach is pairwise preference, asking which of two videos tells a better story or exhibits fewer artifacts.

  3. Implementing the Evaluation Judge in CI 17:27

    The evaluation judge (e.g., Qwen3-VL) is used as a regression gate within the CI pipeline, ensuring that every AgentX release clears an 'eval wall' calibrated against human scores before reaching users. Catching drift early significantly reduces correction costs.

  4. Training Data Improvement 20:47

    Initial models failed because they scored the 'vibe' (gloss) rather than the substance (story/physics). The fix involved creating a dataset by pairing real, human-generated footage against AI-generated footage to prevent the model from becoming an overfitted 'AI detector.'

Watch on YouTube Full article