The Video Signal

Topic

LLM Inference

All digests tagged LLM Inference

Local AI: Running Nemotron on DGX Spark & Station | Nemotron Labs thumbnail

· 56:06

Local AI: Running Nemotron on DGX Spark & Station | Nemotron Labs

The session details the growing viability of local AI, emphasizing that running Large Language Models (LLMs) on dedicated local hardware like DGX Spark and DGX Station offers significant advantages in sovereignty, privacy, and cost efficiency compared to cloud-only inference. Key technical advancements include the 'Exo backend,' a unified software stack that enables consistent model deployment across diverse hardware (Nvidia and Apple silicon). The discussion also covered advanced scaling techniques, such as Process/Data (PD) Disaggregation, allowing compute-intensive parts of inference to run on one device while memory-intensive parts run on another, maximizing utilization of heterogeneous local compute resources.

Key takeaways

  1. Local AI Value Proposition 2:50

    Local AI provides sovereignty, privacy, and cost efficiency by eliminating per-token cloud costs. For single-user, low-concurrency workloads, local inference can achieve high tokens per second, matching or exceeding the performance of cloud serving setups (Transcript, 0:02:50).

  2. Unified Local AI Stack (Exo Backend) 7:20

    The Exo backend addresses the fragmentation problem in local AI by providing a unified stack that runs across different hardware targets, including Nvidia hardware and Apple silicon (M3 Ultra). This ensures that models can be run consistently regardless of the underlying device (Transcript, 0:07:20).

  3. Advanced Scaling and Hardware Utilization 11:17

    DGX Spark and DGX Station support both horizontal (clustering) and vertical scaling. DGX Station, with up to 750 GB of unified VRAM, is recommended for running very large open-source models (e.g., GLM 5.3) that exceed the capacity of a single Spark unit (Transcript, 1:17:00).

Watch on YouTube Full article

The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI thumbnail

· 14:59

The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

The talk outlines the shift from traditional request-based LLM inference (chat) to complex, task-based agentic inference. Because agents run in long loops (plan, act, observe, repeat) and generate massive context prefixes, standard inference stacks fail. FriendliAI addresses this by presenting an 'inference cloud for agents' built on four pillars: Prefix Caching, hierarchical KV Cache Management, Cache Aware Routing, and Agent Aware Scheduling. This architecture enables open-weight models (like GLM 5.2) to achieve frontier quality at a fraction of the cost of closed models, making complex, multi-step agents economically viable for production.

Key takeaways

  1. The Unit of Work Shifts from Request to Task 5:18

    In chat, the unit is the request (optimizing for low latency). In agents, the unit is the task, which runs autonomously for minutes or hours. Optimization must therefore focus on end-to-end task latency, not single-call latency. (3:18)

  2. Open Weights Achieve Frontier Quality at Lower Cost 2:06

    Open-weight models are now capable of completing complex agentic tasks (e.g., building a tower defense game) at a usable level, rivaling closed models. For the same task, GLM 5.2 (open-weight) was approximately 5.6 times cheaper than Anthropic's Opus 4.8. (2:06)

  3. The Four Pillars of Agentic Inference Cloud 11:42

    To optimize for task latency, the stack relies on: 1) Prefix Caching (computing shared prefixes once); 2) Hierarchical KV Cache Management (across GPU, host, and disk); 3) Cache Aware Routing (sending requests to replicas holding the prefix); and 4) Agent Aware Scheduling (understanding the long-term program context). (7:02)

Watch on YouTube Full article

Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google thumbnail

· 16:07

Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google

This talk addresses the critical issue of unreliable LLM performance benchmarks, arguing that often the failure point is the benchmark harness itself, not the inference server. The speakers introduce `Inference Perf`, a CNCF project, which provides a standardized, high-fidelity solution. This tool solves common pitfalls—such as Python's Global Interpreter Lock (GIL) limitations, client-side latency inflation, and non-reproducible configurations—by offering declarative configuration, robust load generation across multiple processes, and combined client/server telemetry for accurate production-scale measurement.

Key takeaways

  1. Benchmark Harness vs. System Under Test

    Many performance discrepancies are incorrectly attributed to the server when the fault lies with the benchmark harness. Issues include GIL limitations in Python, client-side latency inflation (e.g., up to 58 seconds), and non-deterministic settings (e.g., model temperature set to zero).

  2. Need for High Metric Fidelity

    Production-scale benchmarking requires observing both the system under test and the client's behavior simultaneously. The proposed solution, `Inference Perf`, provides client-side telemetry alongside server metrics to ensure accurate diagnosis.

  3. Standardization of Workloads

    The `LLMD` project provides a published workload catalog defining state-of-the-art inference tasks (e.g., agentic generation, tree of thought, batch summarization) in a standardized, declarative format, improving reproducibility across different tools.

Watch on YouTube Full article

Build A Reasoning Model From Scratch 4: Inference Scaling 1 (Temperature, Top-p, Self-Consistency) thumbnail

· 1:37:12

Build A Reasoning Model From Scratch 4: Inference Scaling 1 (Temperature, Top-p, Self-Consistency)

This session details advanced inference-time scaling techniques used to improve Large Language Model (LLM) accuracy without retraining the base model. Techniques covered include Chain-of-Thought (CoT) prompting, Temperature Scaling, Top-p Filtering, and Self-Consistency (majority voting). The core mechanism involves modifying the text generation function to support diverse sampling, demonstrating that while these methods significantly boost accuracy (e.g., achieving 52% accuracy on Math500), they also dramatically increase computational cost and latency.

Key takeaways

  1. Inference Scaling vs. Training Scaling

    Improving model performance can be achieved by increasing compute either during training (e.g., larger models, more data) or during inference (inference-time scaling). Inference scaling techniques include generating more tokens or using advanced sampling methods like majority voting.

  2. Chain-of-Thought (CoT) Prompting 29:03

    Adding prompts like 'Let's think step by step' is a simple, effective modification that significantly boosts the base model's accuracy by forcing intermediate reasoning steps.

  3. Temperature Scaling and Sampling 1:03:40

    Temperature controls the output diversity by rescaling logits before applying softmax. A lower temperature (e.g., 0.5) increases the peak probability, making the model more deterministic, while a higher temperature increases randomness.

  4. Top-p Filtering

    This technique improves quality by filtering out low-probability tokens, keeping only the most likely tokens whose cumulative probability sum up to a threshold $p$ (e.g., 0.8).

  5. Self-Consistency

    Self-consistency uses majority voting: the LLM generates multiple answers (e.g., 3 or 5) using diverse sampling, and the most frequent answer is selected as the final, most robust result. This method is highly effective for numerical tasks like Math500.

Watch on YouTube Full article

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher thumbnail

· 1:28:12

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

This workshop provides a deep dive into optimizing Large Language Model (LLM) inference at scale. The core challenge is managing the high computational and memory costs associated with LLMs, particularly as context length and concurrent users increase. Solutions are categorized into two areas: model optimizations (e.g., quantization, Grouped Query Attention, Flash Attention) to reduce size/compute, and serving optimizations (e.g., Paged Attention, Continuous Batching, Prefix Caching) to improve throughput and memory efficiency. The session concludes with a comparative analysis of leading inference engines like vLLM and SGLang.

Key takeaways

  1. Inference Cost Drivers 0:05

    The primary pain points are: 1) Memory consumption that grows with context length (due to the KV cache); 2) Time To First Token (TTFT) degradation as input size increases; and 3) Throughput collapse when requests are processed sequentially.

  2. Memory Scaling is Critical 0:10

    The Key-Value (KV) cache memory usage scales significantly: for a Mistral 7B model, the KV size per token is approximately 131 KB. Serving multiple users with long contexts quickly exhausts GPU VRAM (e.g., 80 concurrent users and 4K context can require 42 GB).

  3. Serving Optimizations are Essential 0:24

    To maximize throughput, techniques like Paged Attention (borrowed from OS paging) solve memory fragmentation; Continuous Batching keeps the GPU occupied by processing multiple requests efficiently; and Prefix Caching saves computation when handling similar prompts across different users.

  4. Engine Selection Trade-offs 0:46

    While vLLM and SGLang show no statistical difference on standard workloads, SGLang demonstrated a significant advantage (three to four times better) when handling complex agentic branching workflows.

Watch on YouTube Full article

KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat thumbnail

· 21:48

KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

This talk details advanced strategies for optimizing LLM inference in complex agentic workloads, moving beyond the limitations of steady-state public benchmarks. The focus is on two critical levers: KV Cache-Aware Routing and Prefill/Decode (P/D) Disaggregation. Implementing these techniques—using frameworks like LLMD—significantly improves latency and throughput by managing volatile cache usage and separating compute-bound prefill from memory-bandwidth-hungry decode phases, particularly in the middle concurrency band.

Key takeaways

  1. Agentic Workloads vs. Benchmarks 5:29

    Real-world agentic workloads exhibit chaotic multi-turn interactions (up to 3,000 turns) with high cache hit rates (>90%) and massive input/output ratios (often >100:1), which standard public benchmarks fail to capture [0:00], [3:29].

  2. KV Cache Routing Optimization 10:28

    Implementing KV cache-aware routing (via Endpoint Picker) is a cost-effective optimization, as the token cost difference between cached and uncached tokens can be as high as 10x [5:12]. This helps solve Time to First Token (TTFT) issues.

  3. P/D Disaggregation Benefits 15:28

    Separating prefill and decode into independent, scalable pods prevents 'phase interference'—where a long prefill stalls token generation (decode)—leading to drastically reduced P99 Inter Token Latency (ITL) from ~900ms down to ~100ms [9:28], [10:46].

  4. Prerequisites for PD 17:26

    Effective P/D disaggregation requires an advanced, high-speed network fabric like RDMA or RoCE to facilitate the transfer of KV caches between prefill and decode workers [10:46]. If such a fabric is unavailable, aggregated serving may be preferable.

Watch on YouTube Full article