The Video Signal

Topic

Agentic Systems

All digests tagged Agentic Systems

One Operator, Many Drones: Inside Skydio's Autonomy Stack — Suchet Bargoti, Skydio thumbnail

· 20:48

One Operator, Many Drones: Inside Skydio's Autonomy Stack — Suchet Bargoti, Skydio

Skydio presented its full-stack autonomy solution, demonstrating how drones are evolving from hobbyist tools into critical infrastructure. The system enables large-scale, multi-agent orchestration, allowing a single operator to manage multiple drones performing diverse tasks (e.g., utility inspection, tracking stolen vehicles) across different geographical locations simultaneously. The core technical advancements involve splitting intelligence between the edge (on-drone actions) and the cloud (long-term planning, heavy lifting), utilizing World Models for global path planning, and employing Visual Language Models (VLMs) for agentic, rule-free object tracking and semantic reasoning.

Key takeaways

  1. Drones as Infrastructure 2:00

    Skydio is positioning its drones as critical infrastructure, with thousands of docks deployed across utilities, public safety, and construction sectors. This allows for continuous, reliable operation (day/night, rain/sunshine) and scales beyond the limitations of requiring a dedicated pilot for every incident.

  2. Full-Stack Autonomy Architecture 18:50

    The autonomy stack splits intelligence between the edge (for immediate actions) and the cloud (for heavy lifting and long-term planning). This architecture is designed to maintain high reliability (targeting 99.9999%) while managing vast amounts of data and complex decision-making.

  3. Agentic Orchestration

    The system moves beyond hand-coded rules by using agentic tools. A VLM can receive a high-level command (e.g., 'find a white Jeep') and autonomously access APIs and tools to command a drone's trajectory, enabling 'find and follow' without specific coding for every scenario.

Watch on YouTube Full article

Seattle Spark Hack Winners Livestream Spotlight: LiveKit & Memo thumbnail

· 42:28

Seattle Spark Hack Winners Livestream Spotlight: LiveKit & Memo

This summary covers the NVIDIA DGX Spark Hack winners spotlight, featuring two local AI applications: LifeKit, an offline survival companion, and Memo, a local-first wearable assistant. Both projects demonstrate the capability of running complex, multi-modal AI workflows—including vision, speech, and retrieval—on edge devices like the Acer Veriton GN100. The core technical focus is on achieving reliable, low-latency AI functionality without constant internet connectivity, utilizing advanced NVIDIA models like Neotron and Cosmos.

Key takeaways

  1. Offline AI Capability (LifeKit) 20:00

    LifeKit addresses the critical need for stable information access when internet connectivity fails. It uses local AI and Retrieval Augmented Generation (RAG) against extensive, downloadable knowledge bases (e.g., Wikipedia) to provide guidance on survival topics (first aid, navigation) without relying on external servers.

  2. Selective Memory and Privacy (Memo) 27:30

    Memo is a wearable assistant designed for elderly care, focusing on privacy by only tracking objects explicitly chosen by the user (e.g., keys, wallet). It uses visual evidence and confirmation to save location data, ensuring that the system does not record or store general video feeds.

  3. Concurrent Model Orchestration 22:30

    Both projects rely on running multiple NVIDIA models concurrently (e.g., Neotron, Cosmos, Parakeet, VSSs, NIM, VLM) on the GN100. The primary architectural challenge is orchestrating these disparate tools—from speech transcription to reasoning—into a single, reliable, and deterministic application.

Watch on YouTube Full article

AWS's Marc Brooker: Specs, Not Code, Are the Hard Part thumbnail

· 56:50

AWS's Marc Brooker: Specs, Not Code, Are the Hard Part

The core argument is that the hardest part of modern software development is no longer writing code, but defining the specification—a complete, detailed description of what the system should do. Marc Brooker, VP and Distinguished Engineer at AWS, emphasizes that advanced automation, particularly agentic AI, is making the building and maintenance of reliable implementations increasingly automated. This shifts the engineering focus to specification, rigorous testing, and managing complex system failures, such as metastable failures. Key technical advancements include specialized policy languages (like Dogwood) and advanced memory systems that allow agents to learn from historical postmortems and organizational context.

Key takeaways

  1. The Shift to Specification 33:44

    The most challenging aspect of software development is creating a complete, detailed specification of what is desired. Once this specification is established, the building and maintenance of reliable implementations are expected to become increasingly automated (00:32:14).

  2. The Importance of Postmortems 11:11

    Analyzing thousands of postmortems teaches engineers a 'deep sense of humility' regarding system failures and highlights the need to understand complex failure modes like metastable failures (00:11:23).

  3. Agentic Policy vs. Classic Authorization 20:14

    Classic authorization models are insufficient for autonomous agents. Agentic policy requires a specialized language (like Dogwood) to deterministically define what an agent is allowed to do, and in what order, providing necessary boundaries for trust (00:20:14).

  4. Learning from Failure 11:14

    Agents can be trained to learn from postmortems by using tools to semantically search historical incidents and build procedural memory, allowing them to anticipate and circumvent past failures (01:14:32).

Watch on YouTube Full article

How Lyft Increased Its Agent Resolution Rate by 16% with LangSmith and LangGraph thumbnail

· 3:41

How Lyft Increased Its Agent Resolution Rate by 16% with LangSmith and LangGraph

Lyft addressed the challenge of scaling its customer support agent stack by replacing brittle, deterministic agents with a meta-agent architecture built on LangGraph and LangSmith. This new self-serve platform allows non-engineering personnel (PMs and ops) to deploy new agents via simple configuration and prompting, drastically reducing agent build time from six months to one to two weeks. This accelerated iteration cycle resulted in a 16% increase in the customer resolution rate.

Key takeaways

  1. Shift to Self-Service Agent Platform 2:25

    The team created a platform enabling PMs and ops to build and ship agents using domain knowledge and natural language prompting, minimizing the need for code changes (merely a config change).

  2. Architectural Improvement via Meta-Agent 3:35

    The system utilizes a meta-agent where all sub-agents are registered dynamically as nodes in the meta-agent, simplifying the composition and deployment of new agents.

  3. Significant Operational Gains

    The agent build time was reduced from six months to one to two weeks, allowing engineers to focus on complex, foundational improvements while increasing the overall resolution rate by 16%.

Watch on YouTube Full article

Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium thumbnail

· 19:14

Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium

The talk analyzes the evolution of voice agents, arguing that current real-time voice models are fundamentally half-duplex (either listening or speaking). The core technical challenge is achieving full-duplex communication, which involves modeling overlapping speech (like backchanneling). The speaker, Neil Zeghidour, proposes that the most viable path forward is a hybrid architecture: coupling a small, highly natural, full-duplex speech-to-speech (S2S) interface with a powerful, asynchronous background text LLM to handle all complex reasoning and tool calling. This approach mitigates the inherent trade-off where improving naturalness sacrifices intelligence.

Key takeaways

  1. Evolution of Voice Agents 0:10

    Voice agent technology has progressed through constrained, closed-ended dialogue (Siri, 2011) to open-ended conversational models (OpenAI Voice Mode), and finally to agentic systems capable of real actions (e.g., ordering food).

  2. The Full-Duplex Challenge 12:28

    Human conversation is full-duplex, allowing for overlapping speech and backchanneling (e.g., 'Mhm, yeah'). Current S2S models, even with low latency, are limited by fundamental turn-taking mechanisms, making them feel unnatural.

  3. The Intelligence vs. Naturalness Trade-off 18:20

    There is a fundamental tension: every gain in naturalness (e.g., moving from cascaded STT/LLM/TTS to S2S) requires dedicating model capacity (weights) to audio modalities, which reduces the model's overall intelligence and reasoning capability.

  4. The Hybrid Solution 18:40

    The recommended approach is to split the system: use a small, on-device, full-duplex S2S model for natural conversation flow, while delegating all complex reasoning, tool calling, and agentic capabilities to a separate, powerful background text LLM.

Watch on YouTube Full article

Every step you take, every call you make: the reliable agent stack — Giselle van Dongen, Restate thumbnail

· 20:50

Every step you take, every call you make: the reliable agent stack — Giselle van Dongen, Restate

This talk introduces Restate, an open-source framework designed to provide a flexible, durable foundation for building resilient, long-running, and stateful agentic systems. Restate addresses the infrastructure gap in agent development by handling complex concerns like retry logic, recovery, session isolation, and process cancellation. It operates as a server proxying requests to the agent service, using an event journal to ensure that processes can survive crashes, redeploys, and long periods of suspension (e.g., waiting for human approval) without losing state or consuming serverless execution time.

Key takeaways

  1. Durable Execution and Resilience 5:40

    Restate enables durable execution, allowing an agent process that runs for extended periods (e.g., a week) to crash and restart exactly at the point of failure, rather than starting over. This is achieved by recording all events in a journal.

  2. Modeling State with Virtual Objects 14:00

    Instead of modeling agents as simple workflows, Restate uses 'virtual objects' to model persistent, stateful entities (like a session). These objects have unique IDs and isolated state, allowing multiple agents to interact with a single run without interfering with each other's state.

  3. Advanced Control and Interaction 16:40

    The framework supports advanced control primitives, allowing external processes to signal, inject state into, or completely cancel an already running agent loop. This capability is crucial for complex, multi-agent interactions.

  4. Low Latency via Push Model 19:20

    Unlike traditional workflow orchestrators that poll for new tasks (pull model), Restate uses an event-driven, push model for invocations. This design significantly lowers latency, achieving low latencies (e.g., 45ms p99) even for multi-step workflows.

Watch on YouTube Full article

Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic thumbnail

· 13:21

Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic

The presentation challenges the assumption that all tokens used in agentic systems are fungible. Instead of relying solely on increasing the token budget (brute force), the speakers propose giving tokens specific 'jobs' through advanced strategies. These strategies—Advising, Grading, and Dreaming—allow agents to improve performance and reliability within a fixed budget, particularly when the task requires high precision (e.g., financial analysis). The analysis showed that while increasing the budget improves performance across all methods, specialized strategies significantly reduce the true cost (tokens) required to achieve a perfect, reliable outcome.

Key takeaways

  1. Tokens are not fungible

    The core thesis is that tokens should be assigned specific jobs (e.g., advising, grading, dreaming) rather than being used indiscriminately for simple execution. This allows for more efficient use of computational resources.

  2. Three core agent strategies 3:45

    Strategies include: 1) Advising (splitting an executor from an adviser that provides mid-task guidance); 2) Grading (defining a rubric and using a grader to iterate until a perfect outcome is achieved); and 3) Dreaming (allowing tokens to inspect transcripts and write findings to memory for future runs).

  3. Cost vs. Accuracy in Financial Analysis 11:40

    When evaluating financial tasks (like creating a P&L statement), 80% accuracy is considered useless if 100% accuracy is required. The analysis showed that the baseline 'Execute' strategy required an average of 1.8 million tokens to achieve a perfect answer, while 'Advise' and 'Grade' were significantly more token-efficient.

  4. Optimization depends on the goal

    The choice of strategy depends on the optimization goal: choose 'Advise' for maximum token efficiency, or 'Grade'/'Dream' if maximizing the percentage of perfect, reliable runs is the priority.

Watch on YouTube Full article

Training Agents 4: From reward functions to environments. thumbnail

· 1:13:40

Training Agents 4: From reward functions to environments.

This session details the evolution of agent training from simple functional reward signals to complex, stateful environments. The core concept is the `reset()/step()` contract, which allows agents to interact with a simulated world (the environment) over a sequence of actions. The discussion covers the OpenM framework, which standardizes environment definition, and its integration with TRL (Transformer Reinforcement Learning) using isolated compute environments like Hugging Face sandboxes. This enables training sophisticated agents, such as coding agents (OpenCode), on complex, multi-step tasks while maintaining reproducibility and isolation.

Key takeaways

  1. The Shift to Stateful Environments 1:42

    For agents performing sequences of actions (e.g., tool calls, file edits), the reward signal must come from the environment's state after an action, rather than being a simple Python function evaluated once. This requires adopting the standard `reset()/step()` contract, moving from sparse signals to continuous interaction loops.

  2. The OpenM Ecosystem 5:30

    OpenM provides a standardized, containerized way to define any task as an environment. It packages the task, the runtime (compute backend), and the grading components (verifiers, rewards) into a single, shareable unit that can be deployed on various platforms (e.g., Hugging Face Spaces, Kubernetes).

  3. Advanced Agentic Training Loops 9:10

    Training can be managed by two models: the 'White Box' (where the training framework owns the loop) and the 'Black Box' (where the agent/harness owns the loop). The latter requires a 'capture proxy' to intercept agent calls and send the resulting rollout graph back to the trainer for policy updates.

  4. Reproducibility and Isolation 7:30

    Environments are designed to be self-contained applications, often deployed in isolated compute environments (like HF sandboxes). This ensures that training runs are reproducible and prevents the agent from accessing or manipulating the verifiers or task description to 'hack' the reward signal.

Watch on YouTube Full article

Generative UI... in Python? — Jeremiah Lowin, Prefect thumbnail

· 17:38

Generative UI... in Python? — Jeremiah Lowin, Prefect

This talk introduces Prefect, a Python DSL designed to solve the challenge of delivering rich, interactive UIs within the MCP (Messaging/Communication Protocol) ecosystem. While MCP apps allow tools to bypass the agent and render full HTML/CSS/JS interfaces, the primary user base (Python engineers in enterprises) needed a way to build these UIs without writing complex frontend code. Prefect achieves this by allowing developers to compose UIs using Python context managers and classes, which are then serialized into a JSON protocol and rendered by a React app. A key optimization is streaming the Python code itself over the wire and executing it in a sandbox, resulting in a 70% reduction in size compared to streaming the JSON representation.

Key takeaways

  1. MCP Apps and Agent Bypass

    MCP apps are an extension of the MCP protocol that allows tool results to bypass the agent and be delivered directly to the user as a full UI (HTML, CSS, JavaScript), enabling rich, interactive experiences.

  2. The Prefect Solution 9:03

    Prefect is a Python DSL that allows developers to compose UIs by nesting components using context managers, avoiding the need to write traditional frontend code.

  3. The Core Pipeline 10:37

    The process involves using a Python DSL to build a declarative representation, which is serialized into a JSON protocol. This JSON is then rendered by a React app hosted as the MCP app.

  4. Efficiency Improvement 16:40

    The system was optimized to stream the Python representation of the UI over the wire and execute it in a sandbox, which was found to be approximately 70% smaller and more efficient than streaming the full JSON protocol.

Watch on YouTube Full article

Tethered: Our Agents Are Us — Shu Fang, Two Sigma thumbnail

· 21:10

Tethered: Our Agents Are Us — Shu Fang, Two Sigma

Two Sigma implemented a framework allowing every employee to run cloud agents using their own unique user identity, addressing the challenges of permissions drift and maintaining security in a highly regulated environment. The solution leverages existing Kubernetes infrastructure (dedicated namespaces per person) and introduces two critical guardrails: propagating a trace header for full action provenance, and utilizing Google's web grounding for enterprise—a restricted search index that eliminates external egress vulnerabilities while accepting a data freshness constraint of up to 24 hours.

Key takeaways

  1. Running Agents as User Identity 2:00

    By running agents with the user's exact identity, the system bypasses conventional constraints like permissions drift and licensing issues associated with separate machine identities. This capability was supported by pre-existing infrastructure: a Kubernetes namespace per individual in every region, where automated jobs already ran using the user's identity via a sidecar mounting mechanism.

  2. Ensuring Action Provenance (Attribution) 8:37

    To differentiate between actions taken by the human and those performed by the agent, a dedicated header is propagated throughout the system. This trace ID allows for full provenance tracking, enabling the replay of the entire chain of actions leading to an end result, which is superior to simple identity verification.

  3. Securing Web Access with Grounding 9:18

    To mitigate risks like exfiltration and prompt injection from open web access, the firm adopted Google's 'web grounding for enterprise.' This service provides search and fetch capabilities within the internal VPC network boundary, while blocking native tools (e.g., Brave web browser) to ensure all requests route through the controlled index.

Watch on YouTube Full article

Koray Kavukcuoglu on frontier models, coding agents, and building AGI thumbnail

· 26:47

Koray Kavukcuoglu on frontier models, coding agents, and building AGI

Google DeepMind SVP Koray Kavukcuoglu discusses the ambitious journey toward Artificial General Intelligence (AGI), emphasizing that success relies on moving models from simple coding capabilities to full software engineering and agentic workflows. The discussion highlights the continuous progress of the Gemini model family (e.g., 3.7, Flash) through parallel research tracks and stresses that real-world user interaction is critical for guiding development toward AGI.

Key takeaways

  1. AGI lacks a definitive test or benchmark 14:53

    There is no single standardized test to determine if an AI has reached AGI; progress is measured by the overall journey and capability build-up, not a sudden threshold. (08:53)

  2. The focus shifted from coding to software engineering 2:48

    A major breakthrough in model development was understanding that true intelligence requires more than just writing code; it involves mastering the full scope of 'software engineering,' including working with tools and functions, effectively turning the model into an agent. (02:48)

  3. Gemini 4 is positioned as a major research milestone 3:17

    The team announced Gemini 4 as the most ambitious pre-training run to date, representing a significant step in combining multiple learnings and architectural improvements into one model. (03:17)

  4. User interaction is the guide for AGI development 11:59

    The path to building AGI depends heavily on continuous user interaction and feedback, whether users are performing daily tasks (e.g., emails) or conducting advanced scientific research. This usage spectrum guides problem-solving efforts. (11:39)

Watch on YouTube Full article

How do you diffuse AI into the real world? — Varun Shenoy, Long Lake thumbnail

· 17:46

How do you diffuse AI into the real world? — Varun Shenoy, Long Lake

The deployment of advanced AI agents into real-world service industries is not merely a technological challenge but an operational one. Drawing parallels to the adoption of electricity and Ford's assembly line, the speaker argues that technology diffusion takes generations. Long Lake addresses this by acquiring and operating services businesses (e.g., property management) rather than selling software. Their approach focuses on building AI agents that move beyond simple 'co-pilots' to become autonomous 'co-workers,' leveraging proprietary ground truth data collected from messy, real-world tasks—a process requiring deep, physical co-design with the industry.

Key takeaways

  1. AI Diffusion Takes Generations 1:30

    The adoption of general-purpose technologies (GPTs) is slow. Just as electricity took decades to fully integrate into industries like Ford's, AI requires massive operational shifts—ripping out old processes and retraining staff—to achieve full diffusion. [1:30]

  2. The Value of Owning the Outcome 2:36

    Long Lake does not sell AI software; they acquire and operate services businesses (e.g., HOA, architecture). By being the operator/owner, they bear the risk when the AI fails, ensuring deep integration and accountability that external vendors cannot match. [2:36]

  3. The Progression from Co-pilot to Co-worker 6:18

    AI agents must progress through stages of autonomy: Co-pilot (simple RAG chatbot) $ ightarrow$ Synchronous Agent (real-time, two-way interaction) $ ightarrow$ Asynchronous Agent (background work, external triggers) $ ightarrow$ Long-running Agent $ ightarrow$ AI Co-worker (proactive partner). Achieving the co-worker requires earning the right to do more through iterative field deployment. [6:02]

Watch on YouTube Full article

Cisco & Stanford on Why Skills Are the New Code thumbnail

· 9:44

Cisco & Stanford on Why Skills Are the New Code

The industry is shifting from viewing software development around explicit code and implementation toward one centered on high-level intent and 'skills.' This paradigm requires a layered agent stack (models, tools, context, harnesses) that must be managed rigorously. Experts highlight that skill sprawl leads to failure through overlap, drift, and lack of activation visibility. Crucially, the consensus is that achieving business value relies less on deploying increasingly powerful frontier models and more on sophisticated context engineering and centralized management of skills.

Key takeaways

  1. Skills as the New Code Paradigm 0:14

    Software development is transforming from revolving around code/implementation to revolving around intent and instructions. Skills must be treated as first-class citizens, not just configuration files (Guy Podjarny).

  2. Three Failure Modes of Skill Sprawl 3:29

    Skill sprawl negatively impacts teams through: 1) Overlap (multiple isolated implementations achieving the same outcome); 2) Drift (teams using outdated versions of skills); and 3) Lack of Activation (no visibility into whether a skill is actually being used by agents or humans).

  3. Context Engineering Beats Model Size 6:59

    For achieving business value, smarter context engineering is more critical than deploying the most advanced model. Mid-tier models (e.g., Sonnet, GPT medium reasoning) are often sufficient when provided with proper context and structured skills.

  4. Instruction Following Leakage 8:48

    Empirical testing involving 500 skills across 1,000 tasks revealed that over half (55%) of the time, models followed a skill's instructions even when the skill was not loaded. This suggests valuable information is already encoded in model weights.

Watch on YouTube Full article

Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards — Dan Bjornn, Lease End thumbnail

· 16:39

Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards — Dan Bjornn, Lease End

The video argues that relying heavily on fine-tuned LLMs for production applications can create 'calcification tax'—a form of technical debt characterized by rigid architecture and complex maintenance cycles. The speaker details migrating from a fine-tuning approach to an agentic framework built on skills, prompts, and context. This shift drastically reduced the time required to deploy fixes (from weeks/days to under an hour), improved accuracy, maintained model agnosticism, and ultimately lowered total operational costs.

Key takeaways

  1. The Calcification Tax 10:04

    Fine-tuning creates technical debt by locking systems into specific models and architectures. The complexity of retraining—which involves gathering examples, synthesizing data, manual validation, and iterative fixing—is costly and slow (up to a week per cycle).

  2. Agentic Frameworks Over Fine-Tuning

    The rebuild swapped the fine-tuned model for skills, prompts, and context on a model agnostic framework. This allowed fixes to ship by simply uploading files to an S3 bucket.

  3. Cost vs. Accuracy Tradeoff

    While the new agentic approach increased cost per message (due to using better models), overall total cost decreased significantly because the time spent on maintenance and retraining was drastically reduced.

Watch on YouTube Full article

What should security leaders do with AI? They don’t know. thumbnail

· 29:15

What should security leaders do with AI? They don’t know.

Cybersecurity leaders face 'AI decision fatigue' due to the rapid pace of AI adoption and emerging threats. To mitigate this paralysis, experts recommend starting AI deployment with low-risk, repetitive tasks (L1/L2 alerts) or by arming red teams. The discussion highlighted sophisticated new attack vectors like Ghostjacking—which poisons trusted data sources (logs, alerts)—and questioned the reliability of AI in patching vulnerabilities. Overall, successful AI integration requires a 'human-in-the-loop' approach and strict adherence to Zero Trust principles, particularly regarding agent permissions.

Key takeaways

  1. Start AI deployment with repetitive tasks

    Initial AI adoption should focus on automating predictable, high-volume activities (e.g., triaging L1/L2 alerts or vendor risk assessments) to reduce staff fatigue and build confidence in the technology.

  2. Red Teaming is key for defensive AI use 17:25

    Equipping red teams with advanced tools allows them to simulate sophisticated threat actor behavior, enabling defenders to proactively develop countermeasures against emerging attack methods.

  3. Ghostjacking targets trusted data sources 17:25

    A new attack method involves sneaking malicious prompts into highly trusted systems like logs, alerts, and error reports. This is a sophisticated evolution of prompt injection that bypasses initial security controls.

  4. AI patching requires human oversight

    Research indicates AI-generated patches (e.g., using GPT 5.5) have a low success rate, often failing to solve the underlying vulnerability or creating new issues. Human validation and partnership are crucial.

Watch on YouTube Full article

Building an Agentic Video Editor for Mass Consumer — Ekaterina Deyneka, Reelful thumbnail

· 12:45

Building an Agentic Video Editor for Mass Consumer — Ekaterina Deyneka, Reelful

Reelful addresses the complexity of video editing by implementing an agentic workflow that automates content creation from raw footage. The system accepts user media and directional prompts, utilizing a multi-stage pipeline—including Media Understanding, Creative Planning, and execution within a sandboxed environment—to generate polished videos. Architecturally, the process is compared to agentic app building, but with the added complexity of judging optimal cuts and structure from messy, real-world footage.

Key takeaways

  1. Agentic Video Editing Definition 2:09

    The user provides raw media and context/directions (e.g., 'add captions,' 'add music'). The agent then autonomously finds usable moments, assembles the clip, and generates supplementary assets like captions, music, voiceovers, and B-roll.

  2. Architectural Parallelism 5:33

    From an infrastructure standpoint, agentic video editing is structurally similar to agentic app building. Both require a prompt/input (media + prompt for video) and execute within a remote 'sandbox' environment by an agent equipped with tools and skills.

  3. The Challenge of Editing vs. Generating 6:50

    Editing real footage is significantly more complex than generating content from scratch, as the agent must actively judge which moments to keep or omit and organize messy, incomplete material into a polished result.

  4. The Full Agentic Pipeline 8:50

    The process involves: (1) Media Understanding/Transcription $ ightarrow$ (2) Creative Plan generation (for user approval) $ ightarrow$ (3) Sandbox execution by the agent (using skills like cut rules, font pairings, B-roll generation) $ ightarrow$ (4) Remotion composition $ ightarrow$ (5) Verification Layer (to ensure renderability).

Watch on YouTube Full article

Building a Doom-Like World to Explore Agentic Systems - Alexander Chernov - NDC Toronto 2026 thumbnail

· 52:42

Building a Doom-Like World to Explore Agentic Systems - Alexander Chernov - NDC Toronto 2026

This talk presents an architectural framework for building complex agentic systems using a modified Doom-like game engine as a controlled testbed. The core concept is treating agents as 'semantic mirrors' of the game world state, allowing non-player characters (NPCs) to act autonomously while maintaining strict observability and reproducibility. The architecture emphasizes decoupling AI reasoning from the game loop via specialized components like the MCP Gateway, enabling real-world application of simulation techniques in fields such as pharmaceutical R&D.

Key takeaways

  1. Agentic Systems Architecture 2:00

    The system models agents as autonomous entities that perceive the environment and make decisions. The architecture is designed to be observable, attributable, and reproducible through structured world state changes (the 'semantic mirror').

  2. Two-Tiered Agentic Vision 4:20

    To manage latency, a two-tier vision system is implemented: a fast, deterministic observer swarm (7 Hz) for basic tracking, and a slower, LLM-powered tier using 'Lenses' to extract complex semantic information from the environment.

  3. Architectural Components 5:40

    Key components include the Policy Guard system (defining what agents can/cannot do), the MCP Gateway (Model Context Protocol) for external integration, and a Semantic Cache (Mosquito Dog) to reduce latency and cost by caching LLM responses.

  4. Reproducibility and Validation 7:50

    The design ensures determinism through fixed control loops (e.g., 35 ticks per second), state machine transitions, and structured logging of events (JSONL). This allows for full replay and behavioral regression testing.

Watch on YouTube Full article

Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain thumbnail

· 20:02

Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

The continuous improvement of autonomous agents requires shifting focus from code determinism to data mining agent traces. The speaker argues that observability and continual learning are fundamentally linked: an agent's actions in an environment generate a trace record that serves as the substrate for all future improvements. Techniques like harness engineering, distillation (SFT), and analyzing counterfactual traces allow developers to systematically improve agents at lower costs than relying solely on frontier models.

Key takeaways

  1. Shipping is the First Step

    To gather data for improvement, an agent must be deployed into a real-world environment (shipping it). This process generates valuable trace data from tool calls, API usage, and CLIs.

  2. Observability = Continual Learning 4:04

    There is a tight coupling between observability and continual learning for agents. Both require comprehensive traces—the record of actions taken in the environment—to allow the agent to update its internal knowledge or definition.

  3. The Value of Traces 6:00

    Traces capture fine-grained behavior that simple pass/fail benchmarks miss. They are crucial for proving counterfactuals (e.g., comparing GPT 5.5 vs. GLM 5.2) and understanding how agents behave at a granular level.

  4. Improvement Loop Strategy 13:00

    For agent improvement, the recommended loop is: Start with Harness Engineering (fast feedback, ~2 minutes) $\rightarrow$ Saturate this ceiling $\rightarrow$ Fine-tune the model to break through it $\rightarrow$ Return to Harness Engineering.

  5. The Future of Data 17:00

    Agent activity will generate data at an exponential rate, potentially eclipsing all human-produced data in history. Managing this requires building systems that can efficiently mine and process traces.

Watch on YouTube Full article

Evolution of agentic surfaces — Gagan Bhat & Isabella Kai He, Anthropic thumbnail

· 31:24

Evolution of agentic surfaces — Gagan Bhat & Isabella Kai He, Anthropic

The evolution of agentic surfaces (from the Messages API to Claude Managed Agents) addresses the challenge that static harnesses become limiting factors as underlying LLMs improve. The core architectural shift is decoupling the agent's 'brain' (the reasoning loop) from its 'hands' (the tool execution environment/sandbox). This separation significantly improves reliability, allows for parallel setup, and boosts performance by achieving up to 60% faster time-to-first-token at P50. Managed Agents provide production-grade infrastructure—including session logging, credential vaults, and self-hosted sandboxes—enabling developers to focus on defining the agent's task and context rather than managing complex operational overhead.

Key takeaways

  1. Harnesses Encode Stale Assumptions 11:42

    As models improve (e.g., Opus 4.5 eliminating 'context anxiety'), fixes built into older harnesses become pure overhead, adding latency or discarding cache incorrectly. Managed Agents are designed to be agile and adapt to model evolution.

  2. Decoupling Brain from Hands 20:00

    Separating the agent loop (brain) from the tool execution environment (hands/sandbox) improves reliability, allowing the brain to resume from a durable session log even if the hands fail. It also allows model reasoning to start immediately while container setup runs in parallel.

  3. Self-Improving Agents via Dreaming 27:30

    The system utilizes durable session logs and memory state, which can be fed into a periodic batch process called 'dreaming.' This process extracts new insights to automatically update the agent's memory, making subsequent sessions smarter.

Watch on YouTube Full article

The State of Model Routing — NVIDIA, Cognition, OpenRouter thumbnail

· 48:17

The State of Model Routing — NVIDIA, Cognition, OpenRouter

The discussion explores the critical field of model routing in a multi-modal AI landscape, emphasizing that simply sending tasks to the best-benchmarked model is fragile. Solutions involve sophisticated orchestration systems (like Cognition's Fusion) that use cheaper models for implementation while leveraging expensive frontier models for high-level planning and decision-making. Key technical challenges include managing context across multiple agents, minimizing costs through KV cache efficiency (e.g., using sidekick agents), and ensuring model reliability when tasks move from in-distribution to out-of-distribution domains.

Key takeaways

  1. Model Routing is an Orchestration Problem

    Effective AI systems require more than just selecting a single best model; they need robust orchestration that can handle the complexity and changing nature of tasks (e.g., starting as a question, becoming a feature request, then live debugging).

  2. Cost-Efficiency through Delegation 3:35

    Advanced routing allows expensive frontier models to handle planning and decision-making, while cheaper mini-models execute the bulk of the work. This approach can significantly reduce costs (e.g., Cognition claims a 40% cost reduction for Fable-level intelligence).

  3. Context Management is Crucial for Cost Control 6:30

    Using sidekick agents with continuous running context (keeping the KV cache warm) is more cost-effective than traditional main agent/sub-agent systems, as it drastically reduces costs associated with cached tokens.

  4. The Danger of Naive Routing 7:30

    Relying solely on task type for routing is fragile. The complexity and nature of a task change over time, requiring the system to maintain frontier intelligence presence even when delegating work.

  5. Local vs. Cloud Inference Economics 21:45

    Self-hosting models offers greater control over cost dynamics and context management (e.g., setting custom cache lifetimes), fundamentally changing the economics compared to relying solely on API providers.

Watch on YouTube Full article