# New Model: Inkling by Thinking Machine on Hugging Face

## Executive summary

Thinking Machines announced Inkling, a massive open model with nearly one trillion parameters. It is designed as a natively multimodal architecture, accepting image, text, and audio inputs simultaneously. Key technical features include a Mixture-of-Experts (MoE) structure (975B total / 41B active params), a 1M token context window, and advanced deployment support across multiple frameworks like `transformers`, SGLang, vLLM, and `llama.cpp`. The model is available in BF16 and NVFP4 formats, facilitating high-performance inference on various hardware setups.

## Key takeaways

- Multimodal Capability & Scale: Inkling is a true multimodal model that natively processes image, text, and audio inputs using a single architecture. It boasts an immense 1M token context window and was trained on 45T tokens.
- Architectural Innovations: The model utilizes relative attention (replacing RoPE) and a specialized SConv layer to efficiently aggregate hidden states, making it highly efficient for multimodal tasks compared to previous models.
- Deployment Flexibility: Inkling supports multiple deployment paths: BF16 (requiring ~2TB VRAM) and NVFP4 (600GB VRAM). It provides day-zero support for `transformers`, SGLang, vLLM, and `llama.cpp` (including GGUF quants via Unsloth), enabling diverse inference environments.

## Technical details

- Model Architecture: The model uses a Mixture-of-Experts (MoE) design with 975B total parameters and 41B active parameters across 256 experts. It incorporates relative attention, which was noted as being naturally suited for variable-length sequence processing in RL contexts.
- Quantization & Inference: The model is available in BF16 and NVFP4 formats. For local deployment, GGUF quants are provided via Unsloth. Benchmarks showed that even at the one-bit level (270GB storage), it maintained high generation quality and achieved 40 tokens/second on eight H100s using `llama.cpp`.
- Training & Fine-Tuning: The model is designed for post-training customization via the Tinker platform. For fine-tuning, users can leverage the `gold` trainer to perform knowledge distillation, allowing mixing between different tokenizers (e.g., distilling to a Quant or Gemma 4 model).

## Practical implications

- The model's support for multiple inference engines (vLLM, SGLang, llama.cpp) simplifies deployment planning across different compute environments.
- Build engineers can leverage the NVFP4 format to significantly reduce VRAM requirements compared to BF16, making large-scale deployment more feasible on constrained hardware.
- The integration of agentic capabilities (via Pi harness demos) means that model pipelines must now account for complex reasoning loops and code generation steps, not just simple text completion.

## Topics

Large Language Models, Multimodality, Mixture-of-Experts (MoE), Quantization, Inference Optimization, Reinforcement Learning (RL), Thinking Machines Inkling Model (BF16), Thinking Machines Inkling Model (NVFP4), GGUF Quants (Unsloth), Full Blog Post

Source: https://www.youtube.com/watch?v=EOK_haQSMIA