From Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam
Sarvam introduced a 3 billion parameter (3B) State-Space Vision Language Model (VLM) designed for document AI, achieving state-of-the-art (SOTA) performance while being small enough to run on a single GPU. The model is notable for its use of a State Space Model (SSM) backbone instead of a traditional Transformer, which efficiently handles the quadratic complexity of long documents. The training process is a sophisticated four-stage curriculum, covering 13 trillion tokens across English and 22 Indian languages, and includes a proprietary data engine and machine-checkable reinforcement learning (RL) for scalable OCR correctness.
Key takeaways
-
SOTA Document AI Performance
The 3B parameter model achieves SOTA in document AI, claiming to beat models 100 times larger in size.
-
Architectural Innovation (SSM)
3:30
The model uses an SSM backbone instead of a Transformer to manage the high compute costs associated with quadratic attention ($L^2$) when processing long documents (up to 10,000 visual tokens).
-
Comprehensive Training Curriculum
6:00
Training involves four stages: 13T token pre-training (text), 300M image-text pairs (continual pre-training), 100M OCR samples (SFT), and RL using machine-checkable unit tests for rewards.
-
Sovereign Capability
2:30
The entire model building, data, compute, and training were done end-to-end in India, addressing the lack of machine-readable data for Indian languages.