New Model: Inkling by Thinking Machine on Hugging Face
Summary
Thinking Machines announced Inkling, a massive open model with nearly one trillion parameters. It is designed as a natively multimodal architecture, accepting image, text, and audio inputs simultaneously. Key technical features include a Mixture-of-Experts (MoE) structure (975B total / 41B active params), a 1M token context window, and advanced deployment support across multiple frameworks like `transformers`, SGLang, vLLM, and `llama.cpp`. The model is available in BF16 and NVFP4 formats, facilitating high-performance inference on various hardware setups.
Key takeaways
-
Multimodal Capability & Scale
3:50
Inkling is a true multimodal model that natively processes image, text, and audio inputs using a single architecture. It boasts an immense 1M token context window and was trained on 45T tokens.
-
Architectural Innovations
5:10
The model utilizes relative attention (replacing RoPE) and a specialized SConv layer to efficiently aggregate hidden states, making it highly efficient for multimodal tasks compared to previous models.
-
Deployment Flexibility
7:10
Inkling supports multiple deployment paths: BF16 (requiring ~2TB VRAM) and NVFP4 (600GB VRAM). It provides day-zero support for `transformers`, SGLang, vLLM, and `llama.cpp` (including GGUF quants via Unsloth), enabling diverse inference environments.
Technical details
-
Model Architecture
310s
The model uses a Mixture-of-Experts (MoE) design with 975B total parameters and 41B active parameters across 256 experts. It incorporates relative attention, which was noted as being naturally suited for variable-length sequence processing in RL contexts.
-
Quantization & Inference
680s
The model is available in BF16 and NVFP4 formats. For local deployment, GGUF quants are provided via Unsloth. Benchmarks showed that even at the one-bit level (270GB storage), it maintained high generation quality and achieved 40 tokens/second on eight H100s using `llama.cpp`.
-
Training & Fine-Tuning
750s
The model is designed for post-training customization via the Tinker platform. For fine-tuning, users can leverage the `gold` trainer to perform knowledge distillation, allowing mixing between different tokenizers (e.g., distilling to a Quant or Gemma 4 model).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.