Inference Optimization

Sep 14, 2026 · 9:20

BF16 vs NVFP4 with Nemotron 3.5 Lightning

This technical deep dive compares two model quantization formats, BF16 and NVFP4, using Nemotron 3.5 Lightning. NVFP4 is presented as a highly efficient, low-precision format that significantly reduces memory footprint and increases throughput for inference. While NVFP4 is recommended for token generation, BF16 or full precision checkpoints are advised for model customization, fine-tuning, or training stability. The technology has expanded, allowing NVFP4 to run efficiently on architectures like Hopper and Ampere, not just Blackwell.

Key takeaways

1. NVFP4 for Inference Efficiency

   Using NVFP4 drastically reduces the memory footprint required to store model weights compared to BF16, making deployment on smaller hardware more feasible. It is recommended for high-speed token generation.

2. BF16 for Training Stability 3:56

   While NVFP4 is ideal for inference, BF16 or full precision checkpoints should be used if the developer plans to customize the model, fine-tune weights, or use custom quantization algorithms, as this ensures better training stability.

3. Minimal Accuracy Loss 2:30

   Despite the quantization from BF16 to NVFP4, the accuracy degradation is reported to be very small (e.g., 99.99% retained), thanks to intrinsic properties of model weights.

4. Broad Hardware Compatibility 5:40

   The ability to run NVFP4 checkpoints is no longer limited to the newest hardware (Blackwell); it is now supported across a wide range of GPUs, including Hopper and Ampere.

• NVIDIA Developer
• Quantization
• Model Compression
• Inference Optimization
• GPU Architecture
• Large Language Models
• NVFP4 What Is It?
• NVFP4 vs BF16 Notebook

Watch on YouTube Full article

Sep 6, 2026 · 1:36:42

Build A Reasoning Model Scratch 2: Loading a Base Model, Text Generation, and KV Caching

This session provides an in-depth, hands-on guide to working with Large Language Models (LLMs) from scratch using PyTorch. The process covers loading pre-trained models (e.g., Qwen3), understanding the tokenization and autoregressive text generation process, and critically, implementing advanced optimization techniques. Key focus areas include utilizing KV caching to drastically reduce inference latency and applying `torch.compile` for graph optimization, which are essential skills for deploying high-performance LLM services.

Key takeaways

1. LLMs Generate Text Autoregressively 57:00

   Text generation is not a single step; the model generates one token at a time (autoregressive process). The output of each step is appended to the input context for the next iteration. This iterative nature makes text generation computationally expensive.

2. Greedy Decoding and Token Selection 1:22:01

   The basic method for selecting the next token involves Greedy Decoding, which selects the token with the highest score (using `torch.argmax`) from the model's output logits. The process relies on the tokenizer to convert these IDs back into readable text.

3. KV Caching for Inference Optimization

   To improve performance, KV caching stores intermediate Key and Value tensors computed during attention mechanisms. Instead of recomputing these values in every step, they are retrieved from the cache, significantly reducing computational overhead and improving throughput (e.g., from 4 to 28 tokens/second).

4. Model Compilation with `torch.compile`

   The `torch.compile()` feature optimizes the PyTorch computation graph by fusing operations, reducing overhead and improving execution speed. This is a powerful optimization technique for deployment but requires careful handling due to potential compatibility issues.

• Sebastian Raschka
• Large Language Models (LLMs)
• PyTorch
• Tokenization
• Autoregressive Generation
• KV Caching
• Model Compilation
• Inference Optimization
• Reasoning Book
• LLMs from Scratch book
• Reasoning GitHub repo
• KV cache from scratch article

Watch on YouTube Full article

Aug 3, 2026 · 1:03:21

Stanford CS329A Self-Improving AI Agents | Part 2 | Test-Time Compute Scaling

The lecture details advanced methods for improving Large Language Model (LLM) performance through 'inference scaling' or 'test-time compute scaling,' rather than relying solely on expensive pre-training. Key techniques include repeated sampling (Large Language Monkeys), which shows that coverage follows a predictable power law with the number of samples. The discussion highlights the critical need for robust verification mechanisms to bridge the generation-verification gap, and concludes by introducing the Arkon framework, an architecture search method that optimizes complex inference pipelines using components like Fusion, Critic, and Ranker.

Key takeaways

1. Inference Scaling Paradigm Shift 1:45

   LLM capability can be significantly enhanced at inference time by increasing compute (e.g., repeated sampling) without modifying model parameters or requiring fine-tuning, offering a new paradigm compared to traditional pre-training and fine-tuning.

2. Repeated Sampling Effectiveness 2:40

   By repeatedly querying the same problem (e.g., using Llama 3-8b or DeepSeek), selecting the correct response among candidates, models can achieve performance comparable to larger proprietary models like GPT-4o.

3. The Role of Verification 7:50

   For repeated sampling to be effective, automated verification is crucial. The 'generation-verification gap' describes the large difference between the best possible outcome (Oracle selection) and what can be achieved using simple methods like majority voting.

4. Advanced Scaling Architectures 20:30

   The Arkon framework treats inference scaling as an architecture design problem, optimizing the combination of techniques (e.g., Fusion, Critic, Ranker) to maximize accuracy given a limited compute budget.

• Stanford Online
• Large Language Models (LLMs)
• AI Agents
• Scaling Laws
• Inference Optimization
• Machine Learning Architecture Search
• Large Language Monkeys paper
• SWE-bench
• KernelBench

Watch on YouTube Full article

Jul 16, 2026 · 36:00

New Model: Inkling by Thinking Machine on Hugging Face

Thinking Machines announced Inkling, a massive open model with nearly one trillion parameters. It is designed as a natively multimodal architecture, accepting image, text, and audio inputs simultaneously. Key technical features include a Mixture-of-Experts (MoE) structure (975B total / 41B active params), a 1M token context window, and advanced deployment support across multiple frameworks like `transformers`, SGLang, vLLM, and `llama.cpp`. The model is available in BF16 and NVFP4 formats, facilitating high-performance inference on various hardware setups.

Key takeaways

1. Multimodal Capability & Scale 3:50

   Inkling is a true multimodal model that natively processes image, text, and audio inputs using a single architecture. It boasts an immense 1M token context window and was trained on 45T tokens.

2. Architectural Innovations 5:10

   The model utilizes relative attention (replacing RoPE) and a specialized SConv layer to efficiently aggregate hidden states, making it highly efficient for multimodal tasks compared to previous models.

3. Deployment Flexibility 7:10

   Inkling supports multiple deployment paths: BF16 (requiring ~2TB VRAM) and NVFP4 (600GB VRAM). It provides day-zero support for `transformers`, SGLang, vLLM, and `llama.cpp` (including GGUF quants via Unsloth), enabling diverse inference environments.

• Hugging Face
• Large Language Models
• Multimodality
• Mixture-of-Experts (MoE)
• Quantization
• Inference Optimization
• Reinforcement Learning (RL)
• Thinking Machines Inkling Model (BF16)
• Thinking Machines Inkling Model (NVFP4)
• GGUF Quants (Unsloth)
• Full Blog Post

Watch on YouTube Full article