From Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam
Summary
Sarvam introduced a 3 billion parameter (3B) State-Space Vision Language Model (VLM) designed for document AI, achieving state-of-the-art (SOTA) performance while being small enough to run on a single GPU. The model is notable for its use of a State Space Model (SSM) backbone instead of a traditional Transformer, which efficiently handles the quadratic complexity of long documents. The training process is a sophisticated four-stage curriculum, covering 13 trillion tokens across English and 22 Indian languages, and includes a proprietary data engine and machine-checkable reinforcement learning (RL) for scalable OCR correctness.
Key takeaways
-
SOTA Document AI Performance
The 3B parameter model achieves SOTA in document AI, claiming to beat models 100 times larger in size.
-
Architectural Innovation (SSM)
3:30
The model uses an SSM backbone instead of a Transformer to manage the high compute costs associated with quadratic attention ($L^2$) when processing long documents (up to 10,000 visual tokens).
-
Comprehensive Training Curriculum
6:00
Training involves four stages: 13T token pre-training (text), 300M image-text pairs (continual pre-training), 100M OCR samples (SFT), and RL using machine-checkable unit tests for rewards.
-
Sovereign Capability
2:30
The entire model building, data, compute, and training were done end-to-end in India, addressing the lack of machine-readable data for Indian languages.
Technical details
-
Model Architecture
210s
The model utilizes a State Space Model (SSM) rather than a Transformer. SSMs maintain a single state through the sequence, resulting in linear compute growth, which is critical for long documents (5-10,000 visual tokens) where Transformer quadratic complexity becomes prohibitive.
-
Document Processing Focus
250s
The team took a contrarian bet on block-level OCR, wrapping it in dedicated layout and reading order harnesses, rather than relying on monolithic page-level VLMs.
-
Training Curriculum (4 Stages)
360s
1. Text Pre-training: 13 trillion tokens (English, 22 Indian languages, math, code) to build a strong language prior. 2. Continual Pre-training: 300 million image-text pairs. 3. Supervised Fine-Tuning (SFT): 100 million OCR samples across diverse components (tables, equations). 4. Reinforcement Learning (RL): Uses machine-checkable unit tests (e.g., character error rate, table structure) for scalable rewards.
-
Data Engine and Evaluation
450s
The 'moat' includes a data engine that creates synthetic data for low-resource languages and curated evaluation sets (evals) to ensure measurements are truly meaningful and SOTA.
Mentioned resources
- Sarvam
- Krishna Prasad Srinivasan
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.