# From Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam

## Executive summary

Sarvam introduced a 3 billion parameter (3B) State-Space Vision Language Model (VLM) designed for document AI, achieving state-of-the-art (SOTA) performance while being small enough to run on a single GPU. The model is notable for its use of a State Space Model (SSM) backbone instead of a traditional Transformer, which efficiently handles the quadratic complexity of long documents. The training process is a sophisticated four-stage curriculum, covering 13 trillion tokens across English and 22 Indian languages, and includes a proprietary data engine and machine-checkable reinforcement learning (RL) for scalable OCR correctness.

## Key takeaways

- SOTA Document AI Performance: The 3B parameter model achieves SOTA in document AI, claiming to beat models 100 times larger in size.
- Architectural Innovation (SSM): The model uses an SSM backbone instead of a Transformer to manage the high compute costs associated with quadratic attention ($L^2$) when processing long documents (up to 10,000 visual tokens).
- Comprehensive Training Curriculum: Training involves four stages: 13T token pre-training (text), 300M image-text pairs (continual pre-training), 100M OCR samples (SFT), and RL using machine-checkable unit tests for rewards.
- Sovereign Capability: The entire model building, data, compute, and training were done end-to-end in India, addressing the lack of machine-readable data for Indian languages.

## Technical details

- Model Architecture: The model utilizes a State Space Model (SSM) rather than a Transformer. SSMs maintain a single state through the sequence, resulting in linear compute growth, which is critical for long documents (5-10,000 visual tokens) where Transformer quadratic complexity becomes prohibitive.
- Document Processing Focus: The team took a contrarian bet on block-level OCR, wrapping it in dedicated layout and reading order harnesses, rather than relying on monolithic page-level VLMs.
- Training Curriculum (4 Stages): 1. Text Pre-training: 13 trillion tokens (English, 22 Indian languages, math, code) to build a strong language prior. 2. Continual Pre-training: 300 million image-text pairs. 3. Supervised Fine-Tuning (SFT): 100 million OCR samples across diverse components (tables, equations). 4. Reinforcement Learning (RL): Uses machine-checkable unit tests (e.g., character error rate, table structure) for scalable rewards.
- Data Engine and Evaluation: The 'moat' includes a data engine that creates synthetic data for low-resource languages and curated evaluation sets (evals) to ensure measurements are truly meaningful and SOTA.

## Practical implications

- The model addresses the critical gap in machine readability for Indian languages, which constitute a large, fast-growing market.
- The focus on sovereign model development allows large enterprises (insurers, banks, governments) to digitize sensitive data locally, addressing data sovereignty concerns.
- The model's efficiency (3B parameters, single GPU) makes advanced document AI accessible for deployment in resource-constrained environments.

## Topics

AI, Vision Language Models (VLMs), Optical Character Recognition (OCR), State Space Models (SSM), Natural Language Processing (NLP), Low-Resource Languages, Sarvam, Krishna Prasad Srinivasan

Source: https://www.youtube.com/watch?v=T72nqdC92PM