The Video Signal

Topic

LLM Optimization

All digests tagged LLM Optimization

Self-Compact Pi Agent: ZERO HYPE Agentic Coding Devlog thumbnail

· 30:03

Self-Compact Pi Agent: ZERO HYPE Agentic Coding Devlog

This devlog introduces the concept of a self-compacting Pi Agent, addressing the critical limitation of context window size in long-running autonomous agent systems. The core innovation is giving the agent the ability to autonomously manage its own context by calling a dedicated tool. By implementing three distinct compaction thresholds (Notice, Warning, and Force), engineers can significantly improve agent reliability, reduce operational costs, and enable scalable, out-of-loop agentic coding workflows.

Key takeaways

  1. Self-Aware Context Management 2:00

    Instead of relying on default compaction settings, the agent is given a dedicated tool to monitor its context window and decide the optimal moment to compact its memory, which is crucial for long-running autonomous swarms (e.g., Fable or Astra swarms).

  2. Three-Tiered Compaction Thresholds 4:10

    The system utilizes three distinct thresholds—Notice, Warning, and Force—to provide the agent with a wide gap for natural stopping points, followed by a short gap before the hard cutoff, maximizing the agent's ability to decide when to compact.

  3. Advanced Prompt Engineering for Control 5:50

    Full control is achieved by overriding the default compaction prompts provided by agent decoding tools (like CodeX or Pi Agent). This includes defining a 'note to self' that survives the summary, enhancing the agent's self-correction capabilities.

  4. Out-of-Loop Scalability 7:30

    The self-compacting mechanism is essential for scaling agents from in-loop to out-of-loop operations, enabling reliable, long-horizon work where human intervention is not present.

Watch on YouTube Full article

What's New in Inference Engineering — Philip Kiely, Baseten thumbnail

· 19:09

What's New in Inference Engineering — Philip Kiely, Baseten

This talk reviews the state-of-the-art in inference engineering, focusing on advancements in data center optimization. The speaker identifies three major areas—Quantization, Caching (KV compaction), and Speculation—that are rapidly evolving. A key theme is the blurring line between dedicated training and inference, where optimizations are increasingly derived from dedicated training processes. Significant advancements include the development of diffusion-based speculative decoding (DFlash) and techniques like KV compaction, which aim to maintain near-lossless memory retention for context.

Key takeaways

  1. Two Types of Inference Engineering 3:33

    Two distinct approaches emerged: Local inference (focusing on getting the model working on limited hardware via quantization, distillation, and pruning) and Data Center inference (focusing on getting the model working, then making it less slow, utilizing techniques like KV-aware routing and speculation).

  2. Training for Inference 5:52

    Optimizations are increasingly coming from dedicated training processes, blurring the traditional line between training and inference. This creates a cycle: faster inference generates more data, which trains a better model, leading to even faster inference.

  3. Speculation Advancements 10:50

    The field has moved from simple speculative decoding to advanced methods like DFlash (diffusion for speculation), which predicts multiple tokens (8 or 16) in a single forward pass, achieving a more than 3x improvement in acceptance rate compared to previous methods.

Watch on YouTube Full article

Where RL Will Take Search — Maximilian-David Rumpf, SID.ai thumbnail

· 9:36

Where RL Will Take Search — Maximilian-David Rumpf, SID.ai

The presentation outlines how Reinforcement Learning (RL) is poised to revolutionize search by moving beyond traditional, fixed-pipeline architectures. While current agentic search offers vastly higher quality results (roughly doubling the chance of finding correct documents), it is prohibitively expensive and slow (minutes vs. milliseconds). The proposed solution is training a specialized, highly efficient sub-agent using RL, which can adapt its search strategy on the fly, leading to massive improvements in speed and cost compared to frontier models or classical pipelines.

Key takeaways

  1. RL Enables Adaptive Search 3:40

    Unlike classical pipelines where decisions are fixed at design time, an RL-trained sub-agent can iterate, search, read results, and refine its query until it is satisfied, making it highly adaptive to complex questions.

  2. Significant Performance Gains 8:10

    Training a specialized model using RL results in search quality that is approximately 20 times faster and about 100 times cheaper than using a general frontier model for the same task.

  3. Sub-Agents for Efficiency 8:50

    By passing the searching and thinking process to a dedicated, cost-effective sub-agent, the main agent only processes high-quality results, drastically reducing the computational cost associated with context window pollution.

Watch on YouTube Full article

Accelerate the self-improving AI loop with CoreWeave ARIA thumbnail

· 8:44

Accelerate the self-improving AI loop with CoreWeave ARIA

CoreWeave ARIA is an AI research and iteration agent integrated into Weights & Biases (W&B) designed to accelerate the self-improving AI loop. It addresses common challenges in AI development, such as stalled iteration cycles, massive data volume analysis, and manual dashboard creation. ARIA automates auto-research, analyzes training metrics and agent traces, generates comprehensive reports with suggested next steps, and assists in optimizing LLM prompts and agent performance.

Key takeaways

  1. Automated Auto-Research Loop

    ARIA can conduct auto-research by analyzing recorded training metrics and agent traces to uncover hidden insights. It generates visualization-packed W&B reports and automatically launches follow-up training experiments based on its findings, minimizing manual effort (5:51).

  2. Agent Performance Optimization

    ARIA supports agent development by analyzing production traces and suggesting improvements. It can specifically help refine system prompts and evaluate multiple prompt alternatives using defined datasets to achieve higher quality results at lower latency (7:07).

  3. Comprehensive Workflow Support 2:30

    Beyond research, ARIA handles time-consuming manual tasks like providing advice, generating code, and executing commands, all while supporting concurrent conversations that can continue running in the cloud (2:21).

Watch on YouTube Full article

Ask the Experts: How NeMo Switchyard Helps Agents Select Models | Nemotron Labs thumbnail

· 49:23

Ask the Experts: How NeMo Switchyard Helps Agents Select Models | Nemotron Labs

NeMo Switchyard is an open-source model routing library designed for AI agents to solve the problem of relying on a single monolithic LLM. It automatically routes each agent query or step to the optimal model—selecting from any combination of local/cloud and open/closed models—based on real-time needs, optimizing for accuracy, cost, and latency. The system operates beyond simple request routing by tracking state across multi-turn agentic workflows, making it a critical component for building robust, efficient AI systems.

Key takeaways

  1. System of Models Approach 2:49

    The industry is moving away from the 'one model to rule them all' concept toward a 'system of models,' where multiple specialized models are used for different tasks, improving efficiency and capability (0:02:45).

  2. Agent-Aware Routing vs. Simple Routing 5:18

    Switchyard is more than a simple router; it operates on an agentic workflow, tracking state (e.g., tool calls, message history) across multi-turn sessions to make intelligent model selection decisions (0:04:25).

  3. Optimization and Learning 23:18

    The system treats model selection as an optimization problem. It can learn by analyzing agent traces and behavior, predicting potential errors or resource needs to route proactively and save tokens/time (0:21:58).

  4. Full-Stack Routing Flywheel 6:10

    The roadmap envisions a full 'flywheel' of routing, connecting model selection to inference optimization (via NVIDIA Dynamo) and data privacy/anonymization. This allows for continuous improvement across the entire agent lifecycle (0:06:10).

Watch on YouTube Full article

Hands on with Gemini 3.7 Flash thumbnail

· 4:21

Hands on with Gemini 3.7 Flash

The session details how Gemini 3.7 Flash is being utilized by enterprise leaders (Box, Databricks, Emergent) to build highly scalable and cost-efficient multi-agent systems. The focus shifts from merely using the most advanced frontier model to adopting a 'Minimum Viable Model' approach that prioritizes low latency and efficient token usage, enabling complex workflows in regulated industries like finance and insurance.

Key takeaways

  1. Cost-Effective Scaling via Gemini 3.7 Flash

    Gemini 3.7 Flash allows developers to perform tasks using agents at significantly reduced costs (cited as up to 10x cheaper), providing greater budget for further development and exploration.

  2. Pragmatic AI Adoption

    The industry trend favors optimizing cost while maintaining frontier accuracy, moving away from simply adopting the 'coolest' top-tier model. The goal is finding the best model for the job—reliable, low cost, and fast.

  3. Multi-Agent System Architecture 2:00

    Complex tasks are handled by multi-agent patterns: a main agent handles planning (determining required data), which then dispatches specialized subagents to work on parallel exploration. This requires low latency and efficiency.

Watch on YouTube Full article

FIXING Opus 5: PROOF that Prompt Engineering IS NOT DEAD thumbnail

· 34:03

FIXING Opus 5: PROOF that Prompt Engineering IS NOT DEAD

This guide details advanced prompt engineering techniques designed to transform verbose and overly chatty Large Language Models (LLMs), such as Opus 5, into precise, efficient 'senior engineers.' The core principle emphasized is that the **System Prompt**—which acts as the global law for all interactions—provides vastly more leverage than the User Prompt. By implementing structured patterns like reference points, hard operational boundaries, and custom aliases, users can dramatically improve communication clarity, reduce unnecessary output tokens, and ensure consistent, actionable results across multiple agentic tasks.

Key takeaways

  1. System vs. User Prompts 2:00

    The System Prompt sets the overarching rules for an AI agent's behavior (the 'law'), affecting every single user prompt and response. This is where true leverage lies, as it dictates global communication patterns and helps reduce expensive output token costs dramatically.

  2. Positive and Negative Patterns 7:10

    Define explicit behavioral boundaries by telling the agent exactly what to replicate (positive patterns) and what phrases or behaviors to avoid (negative patterns). Examples include avoiding 'load-bearing,' 'worth stating plainly,' analogies, and excessive em dashes.

  3. Reference Points (D1/R1/F1) 15:50

    Use structured codes (e.g., D1 for Decisions, R6 for Risks, Fs for Findings) to create a shared, quick language with the agent. This prevents repetition and allows instant cross-referencing of complex information within the model's output.

  4. Hard Operational Boundaries 22:00

    To prevent scope creep and over-analysis, explicitly instruct the agent to 'deliver ONLY what was requested.' Boundaries should prohibit widening work into cleanup, refactoring, documentation, or speculating on future requirements.

  5. Aliases (SCR, ELI, REF) 27:30

    Implement short codes as commands within the system prompt that expand into full instructions on demand. Examples include `ELI` (Explain Like I'm...) and `SCR` (Simplify, Compress, Repeat), allowing for micro-scale control over response formatting and detail level.

  6. In-Context Distillation 34:03

    Provide concrete examples of desired ('do') and undesired ('don't') responses directly within the system prompt. This acts as training data, locking in preferred communication patterns for the agent.

Watch on YouTube Full article

How Unify cut its AI agent costs 95% in two weeks thumbnail

· 1:08:24

How Unify cut its AI agent costs 95% in two weeks

Unify's agent platform automates go-to-market tasks by giving sales representatives 'an engineer in their back pocket.' The discussion details how Unify achieved a 90-95% cost optimization within two weeks of launch. Key architectural insights include optimizing for prompt caching hit rates, designing subagents as simple function calls, and implementing robust data handling over tabular data using virtualized Pandas-like interfaces. Furthermore, the talk emphasizes advanced evaluation techniques (DQA sets) and stateful, durable cloud execution environments like Monty to ensure scalability and cost control.

Key takeaways

  1. Cost Optimization via Prompt Caching 30:25

    Achieving a high prompt cache hit rate is critical for cost-effectiveness. The limit is approximately 15 requests per second (RPS) within OpenAI's cache, and developers must implement strategies to maximize hits, as providers won't solve this distribution problem for you.

  2. Subagents are Function Calls 25:50

    Architecturally, Unify treats a subagent not as a complex process, but simply as an arbitrary function call. This simplifies the system and allows for efficient execution patterns like mapping over rows.

  3. The Importance of Planning Steps 38:20

    Adding an initial planning step (e.g., using GPT-4) to the agent workflow significantly improves quality and efficiency by forcing the model to scout multiple potential trajectories before executing.

  4. Durable Cloud Execution

    To run agents in a cloud environment without full VMs, using specialized tools like Monty (a Python REPL that suspends) is crucial for maintaining statefulness and strong tenancy.

  5. Adversarial LLM Judging

    When using an LLM as a judge, it must be from a different model family than the original agent model to avoid 'mode collapse' or groupthink, ensuring an adversarial evaluation.

Watch on YouTube Full article

Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain thumbnail

· 20:02

Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

The continuous improvement of autonomous agents requires shifting focus from code determinism to data mining agent traces. The speaker argues that observability and continual learning are fundamentally linked: an agent's actions in an environment generate a trace record that serves as the substrate for all future improvements. Techniques like harness engineering, distillation (SFT), and analyzing counterfactual traces allow developers to systematically improve agents at lower costs than relying solely on frontier models.

Key takeaways

  1. Shipping is the First Step

    To gather data for improvement, an agent must be deployed into a real-world environment (shipping it). This process generates valuable trace data from tool calls, API usage, and CLIs.

  2. Observability = Continual Learning 4:04

    There is a tight coupling between observability and continual learning for agents. Both require comprehensive traces—the record of actions taken in the environment—to allow the agent to update its internal knowledge or definition.

  3. The Value of Traces 6:00

    Traces capture fine-grained behavior that simple pass/fail benchmarks miss. They are crucial for proving counterfactuals (e.g., comparing GPT 5.5 vs. GLM 5.2) and understanding how agents behave at a granular level.

  4. Improvement Loop Strategy 13:00

    For agent improvement, the recommended loop is: Start with Harness Engineering (fast feedback, ~2 minutes) $\rightarrow$ Saturate this ceiling $\rightarrow$ Fine-tune the model to break through it $\rightarrow$ Return to Harness Engineering.

  5. The Future of Data 17:00

    Agent activity will generate data at an exponential rate, potentially eclipsing all human-produced data in history. Managing this requires building systems that can efficiently mine and process traces.

Watch on YouTube Full article

The misaligned incentives behind AI coding agents thumbnail

· 50:16

The misaligned incentives behind AI coding agents

The conversation details how AI coding agents, exemplified by Devin, are fundamentally changing software engineering workflows. The industry is moving past simply training larger models and focusing intensely on optimizing cost-efficiency (token spend) and speed. Key technical advancements include the 'sidekick' agent architecture for achieving high price performance, developing advanced evaluation metrics like 'mergeability' via Frontier Code, and implementing proactive automation to shift human engineers into decision-making roles rather than routine coding tasks.

Key takeaways

  1. The Shift from Capability to Efficiency 8:36

    As agents mature, the bottleneck is shifting from model training size to running evaluations and managing costs. The focus has moved toward optimizing speed and cost rather than chasing the absolute best-performing frontier model for every task (5:56).

  2. The Role of Mergeability in Evaluation 14:01

    A critical gap in current evaluation benchmarks is 'mergeability'—determining if code, while technically correct, would improve the overall quality or maintainability of a codebase. Cognition developed Frontier Code to address this (8:41).

  3. Cost Optimization via Sidekick Architecture 35:46

    The 'sidekick' agent architecture allows for running both a high-quality, expensive model and a more price-performant model in parallel. This dual approach enables significant cost savings (up to 35% better price performance) without sacrificing quality (21:46).

  4. Proactive Automation and Productivity Guarantees

    Agents are moving from reactive task completion to proactive automation, handling tasks like triaging messages or suggesting fixes. This capability led Cognition to underwrite a $10 million productivity guarantee based on measuring 'productive engineering output' (46:51).

Watch on YouTube Full article

Build Hour: Valuemaxxing with GPT-5.6 thumbnail

· 54:33

Build Hour: Valuemaxxing with GPT-5.6

The session provides a deep dive into 'Value Maxxing' with GPT-5.6, shifting focus from merely maximizing token usage to optimizing AI agents for concrete business outcomes and cost efficiency. Key strategies include selecting the optimal model (Sol, Terra, Luna) based on workload needs, implementing advanced API features like programmatic tool calling and prompt caching, and structuring agent workflows to minimize redundant context processing.

Key takeaways

  1. Shift from Token Maxxing to Value Maxxing 2:30

    Progress should be measured by the value generated (e.g., time saved, quality improved), not just the number of tokens consumed. This requires defining clear outcomes and measurable 'good' for AI agents (evals).

  2. Model Selection Strategy 4:09

    The GPT-3.5 family includes Sol (flagship/complex tasks), Terra (balanced intelligence/cost/latency), and Luna (high-volume, cost/latency sensitive) to optimize for specific workloads.

  3. Optimizing Agent Workflows with Codex 5:28

    For day-to-day coding, starting with `GPT-5.6 Soul` on medium reasoning is often sufficient. Developers can also trade tokens for time using Fast mode or utilize Chronicle to build task memory.

  4. Advanced API Techniques (Harness Optimization) 6:51

    Implement programmatic tool calling (using a JavaScript sandbox) and prompt caching to significantly reduce input token costs and processing time. Persistent reasoning and compaction also boost performance and cache efficiency.

Watch on YouTube Full article