The Video Signal

Topic

AI Agents

All digests tagged AI Agents

How AI Agents, LLMs & APIs Use Real-Time Data at the US Open thumbnail

· 9:46

How AI Agents, LLMs & APIs Use Real-Time Data at the US Open

This video details an AI-infused system at the US Open that analyzes tennis serves by combining specialized APIs with Large Language Models (LLMs) via AI Agents. The core architectural pattern involves specialized backend services handling massive, complex data processing (e.g., biomechanics and joint tracking) and returning structured, actionable scores. The LLM then uses these scores, rather than raw data, to perform high-level reasoning and generate human-readable insights for the user. This approach effectively divides labor between specialized computation and natural language reasoning.

Key takeaways

  1. The Limitation of Raw Data for LLMs

    LLMs, while excellent at general knowledge, struggle with raw, high-volume numerical crunching, such as processing 21 tracked joints at 50 times per second, which generates millions of position values. Raw data is too large and complex for effective reasoning.

  2. The Role of Specialized APIs

    APIs and specialized services are necessary to process raw data streams into structured, meaningful metrics. For the US Open, this process generates two key scores: **Efficiency** (biomechanics, joint separation) and **Effectiveness** (outcome stats like speed and placement).

  3. The AI Agent Workflow 0:07

    An AI Agent acts as an orchestrator, using a defined list of tools (APIs). When prompted, the agent outputs a structured request, which is executed by the API. The API returns the processed data, allowing the LLM to complete the reasoning cycle and answer the user's question.

Watch on YouTube Full article

You’re Not Thinking Big Enough: Rebuilding Food Systems with AI Agents — Cody Menefee, Firecrawl thumbnail

· 18:26

You’re Not Thinking Big Enough: Rebuilding Food Systems with AI Agents — Cody Menefee, Firecrawl

The speaker proposes using AI agents and Large Language Models (LLMs) to automate and optimize rotational grazing for grass-fed livestock. Currently, pasture management is limited by human labor and intuition. The proposed system requires integrating multiple data inputs—including GPS location, drought conditions, and grass height—to allow an LLM to suggest the optimal next paddock for the herd. Key technical blockers include building a comprehensive knowledge base (using tools like Firecrawl), developing a visualization layer for biomass and biodiversity, and achieving open, software-agnostic hardware (open collars).

Key takeaways

  1. The Problem: Labor Bottleneck in Grazing 7:21

    Currently, only 3% of cattle are raised on pasture, primarily because rotational grazing—which requires daily movement of animals, fences, and water—is extremely labor-intensive. Proper grazing requires constantly moving the herd to allow specific areas to rest and recover.

  2. The Solution: AI-Driven Grazing 10:28

    The goal is to replace the farmer's intuitive judgment by feeding an LLM multiple data inputs (GPS location, drought conditions, grass height) to suggest the next best paddock for the herd. This requires a multi-varied analysis, not a deterministic decision.

  3. System Components and Blockers

    Three main blockers must be solved: 1) Building a knowledge base (using Firecrawl to scrape YouTube and research papers into Open Pasture); 2) Creating a visualization layer to measure biomass and biodiversity; and 3) Developing open, non-proprietary GPS collars/APIs for software innovation.

Watch on YouTube Full article

Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex thumbnail

· 21:04

Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex

The talk outlines the evolution of Retrieval-Augmented Generation (RAG) toward 2026, positing that RAG will decompose into an agent harness and a dedicated context layer. The primary challenge remains unlocking the vast, unstructured context locked in enterprise documents (PDFs, Word, etc.). The solution requires a three-layered platform: 1) Parsing into token-efficient markdown; 2) Semantic storage/document management; and 3) Repeatable, specialized workflows (e.g., KYC, invoice processing). The speaker emphasizes that document OCR is a complex problem due to how PDFs are rendered for display, not machine consumption, necessitating hybrid parsing approaches that combine deep binary understanding with advanced Vision-Language Models (VLMs).

Key takeaways

  1. RAG in 2026: Agent Harness + Context Layer

    Naive RAG (chunking, embedding, top-k retrieval) is outdated. Modern agents integrate retrieval complexity into the agent layer, allowing them to reason about the optimal search term rather than relying on fixed top-k retrieval methods. (0:00 - 1:33)

  2. Context Moves Up the Stack 5:11

    The focus is shifting from managing context window overflow to hooking up specialized Micro-Capabilities Platforms (MCP) and skills. Furthermore, defining programs and tasks is moving from code (Python, TypeScript) toward natural English language definitions. (3:11)

  3. Document Parsing Requires Hybrid Approaches 8:35

    Because PDFs are designed for printing (using glyphs and line segments) and not machine reading, document OCR is difficult. The optimal approach combines pipeline-based understanding of file binaries (e.g., Word's custom XML) with VLM-based visual analysis to achieve high accuracy and low cost. (5:15)

  4. The Three-Layer Document Platform 8:35

    A comprehensive document context layer requires three components: 1) Parsing into token-efficient markdown/metadata; 2) Semantic storage (document management for agents); and 3) Repeatable, specialized workflows (e.g., claims, invoices). (5:15)

Watch on YouTube Full article

Ask the Experts: Evaluating Agent Skills | Nemotron Labs thumbnail

· 55:19

Ask the Experts: Evaluating Agent Skills | Nemotron Labs

This session details the evaluation of autonomous AI agent skills using NVIDIA's open-source framework, SkillEvaluator. The system provides a comprehensive, multi-tier pipeline to ensure that skills are safe, reliable, and genuinely useful before deployment. The evaluation process covers static security scanning (detecting risks like prompt injection and data exfiltration), deduplication checks, and live agent testing in a sandbox environment. This framework is critical for establishing quality gates in AI-driven workflows.

Key takeaways

  1. Three-Tier Evaluation Framework 2:00

    Skill evaluation utilizes a three-tier framework: 1) Static/Security checks (using SkillSpector), 2) Deduplication checks against an internal registry, and 3) Live Agent Testing, which assesses if the skill improves task performance in a sandbox environment.

  2. Security Scanning for Agent Risks 4:00

    SkillSpector is a standalone security scanner that checks for agent-native risks, including prompt injection, data exfiltration, trigger abuse, and tool poisoning. It combines static analysis with optional LLM semantic analysis.

  3. Measuring Skill Utility (Skill Lift) 7:30

    The evaluation measures 'Skill Lift' across five dimensions: efficiency, correctness, effectiveness, discoverability, and security. This determines if the skill genuinely helps the agent perform the task better compared to a baseline.

  4. Live Agent Testing Methodology 9:00

    Live agent evaluation runs the agent in a secure sandbox, performing an A/B test by comparing the agent's performance when the skill is available versus when it is restricted. This confirms if the skill is functionally necessary for the task.

Watch on YouTube Full article

Dexter Horthy: Why We Stopped Trusting AI to Write the Plan thumbnail

· 56:12

Dexter Horthy: Why We Stopped Trusting AI to Write the Plan

The discussion explores the shift in software development from writing code to managing 'software factories' powered by AI agents. The central thesis is that while AI agents can automate much of the implementation, the primary value shifts to defining and codifying *intent* (specs) and *preferences* (taste). The speaker argues that the process of continuous improvement—building the factory itself—is more critical than the act of reviewing individual code pull requests. Human review, therefore, evolves from checking syntax to verifying high-level architectural intent and system constraints.

Key takeaways

  1. The Spec is the New Code 3:44

    The industry trend is moving toward treating specifications (specs) as the primary, verifiable, and executable artifact. This approach aims to capture the full intent of a feature, which can then be compiled into code, rather than relying on the code itself as the source of truth.

  2. Context Engineering and the 'Dumb Zone' 10:03

    Context engineering is crucial for effective agentic development. Early models exhibited a 'dumb zone' where performance degraded significantly when the context window exceeded a certain token count (e.g., 100,000 tokens), emphasizing the need for intentional context management.

  3. The Value of the Software Factory 28:23

    A 'software factory' is a system that automates the entire development lifecycle (planning, building, reviewing, rolling out). The goal is to shift focus from fixing individual bugs to continuously improving the factory's processes and skills, thereby increasing overall velocity.

  4. The Persistence of Human Review 53:52

    While AI is powerful, the speaker asserts that there will always be 'alpha in reviewing something.' Human review will shift from checking code correctness to verifying high-level architectural decisions, business logic, and unique organizational 'taste' that models cannot inherently replicate.

Watch on YouTube Full article

AI Agents Aren't the Revolution. They're the Catalyst! thumbnail

· 10:14

AI Agents Aren't the Revolution. They're the Catalyst!

While AI agents are widely discussed, the speaker argues that they are not the revolution, but rather a powerful catalyst accelerating fundamental, long-lasting improvements across the technology ecosystem. The most significant shifts are occurring in data modernization, system interoperability, API standardization, and the democratization of technical expertise, which are foundational improvements that will persist regardless of the future of agents.

Key takeaways

  1. AI Agents as Catalysts

    The lasting impact of AI is not the agents themselves, but the systemic improvements they force, including data unification, system design modernization, and enhanced digital literacy.

  2. Data Layer Modernization

    AI adoption forces organizations to overhaul data foundations, making data more accessible, searchable, understandable, and reusable by breaking down silos.

  3. System Interoperability and APIs 5:09

    Agents expose system weaknesses, driving the need for APIs that are predictable, well-documented, and machine-readable. This accelerates the shift toward seamless, cross-system connectivity (e.g., MCP, A2A).

  4. Outcome-Based Thinking

    The rise of powerful agentic coding IDEs lowers the barrier to entry, shifting the focus from 'how' (implementation) to 'why' (defining goals and solving high-level problems).

Watch on YouTube Full article

Netlify's Dana Lawson: 'We Ain't Precious No More' thumbnail

· 10:06

Netlify's Dana Lawson: 'We Ain't Precious No More'

The landscape of software development is shifting from developer-centric to builder-centric, driven by AI agents. While agents enable non-technical users (Product Managers, designers) to open pull requests (PRs) and build applications, this transition introduces new challenges. Agents can fail by missing crucial product or design context, even when passing automated CI checks. Successful adoption requires platforms to be designed for these diverse 'builders' and necessitates that Product Managers evolve into 'agent orchestrators' who define the system's boundaries and ensure proper human control planes.

Key takeaways

  1. Agent Failure Due to Context Loss 2:24

    An agent, despite having access to skills, CI requirements, and passing automated checks, can fail by using a generic component (e.g., a generic React button) instead of a specific, context-aware component that holds critical requirements like accessibility patterns. (Marc Sloan, 00:00:24)

  2. Non-Technical PR Merge Metrics 3:26

    Across hundreds of organizations, 74% of PRs opened by non-technical individuals get merged, and 84% of those merge without any developer needing to push follow-up commits. (Tammuz Dubnov, 00:03:26)

  3. The Builder Persona Shift 5:21

    The platform is no longer built solely for developers. The rise of agents means that anyone—therapists, students, small business owners—can build, making the builder persona far broader. (Dana Lawson, 00:05:41)

Watch on YouTube Full article

The Hidden 50% Drop in AI Agents Following Your Rules thumbnail

· 29:18

The Hidden 50% Drop in AI Agents Following Your Rules

The increasing complexity of multi-agent AI coding systems has led to a critical loss of control, evidenced by a reported 50% degradation in agents' adherence to static instruction files like `AGENTS.md` and `CLAUDE.md` [00:14:28]. The talk argues that traditional agile rituals are being replaced by structured, technical controls: detailed specifications (specs), automated verification steps, and advanced merge tactics (like merge queues). To maintain control, developers must move beyond plain text instructions and adopt structured rule sets, such as those used by CodeRabbit, which force adherence across different models.

Key takeaways

  1. 50% Drop in Agent Adherence 10:28

    Baz's data shows a severe, almost overnight, degradation in the usage of static instruction files (`AGENTS.md`, `CLAUDE.md`) by coding agents, suggesting that model releases can break steering capabilities [00:14:28].

  2. The Shift from Rituals to Structure 19:12

    The bottleneck in software development has shifted from human capacity (PR bottleneck) to system consistency. The process is now being governed by three technical pillars: detailed specs (written in Markdown or linked to issues), automated verification, and advanced merge tactics [00:19:32].

  3. Structured Rules Outlast Plain Files 24:20

    Structured rule sets (e.g., CodeRabbit's JSON rule set) are significantly more effective at forcing agent adherence than plain instruction files (`CLAUDE.md`) because they provide a stronger, more consistent constraint across models [00:23:40].

  4. Long-Horizon Tasks are More Consistent 26:30

    While short tasks show high variability, long-horizon code sweeps demonstrate a larger likelihood of agents adhering to correct instructions due to the sheer number of turns and iterations, though users currently prefer faster, shorter loops [00:23:40].

Watch on YouTube Full article

The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI thumbnail

· 14:59

The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

The talk outlines the shift from traditional request-based LLM inference (chat) to complex, task-based agentic inference. Because agents run in long loops (plan, act, observe, repeat) and generate massive context prefixes, standard inference stacks fail. FriendliAI addresses this by presenting an 'inference cloud for agents' built on four pillars: Prefix Caching, hierarchical KV Cache Management, Cache Aware Routing, and Agent Aware Scheduling. This architecture enables open-weight models (like GLM 5.2) to achieve frontier quality at a fraction of the cost of closed models, making complex, multi-step agents economically viable for production.

Key takeaways

  1. The Unit of Work Shifts from Request to Task 5:18

    In chat, the unit is the request (optimizing for low latency). In agents, the unit is the task, which runs autonomously for minutes or hours. Optimization must therefore focus on end-to-end task latency, not single-call latency. (3:18)

  2. Open Weights Achieve Frontier Quality at Lower Cost 2:06

    Open-weight models are now capable of completing complex agentic tasks (e.g., building a tower defense game) at a usable level, rivaling closed models. For the same task, GLM 5.2 (open-weight) was approximately 5.6 times cheaper than Anthropic's Opus 4.8. (2:06)

  3. The Four Pillars of Agentic Inference Cloud 11:42

    To optimize for task latency, the stack relies on: 1) Prefix Caching (computing shared prefixes once); 2) Hierarchical KV Cache Management (across GPU, host, and disk); 3) Cache Aware Routing (sending requests to replicas holding the prefix); and 4) Agent Aware Scheduling (understanding the long-term program context). (7:02)

Watch on YouTube Full article

Hugging Face's MCP Server: Only 62K of 10M Calls Matter thumbnail

· 9:44

Hugging Face's MCP Server: Only 62K of 10M Calls Matter

The video discusses the significant overhead and limitations inherent in current AI agent protocols, particularly the Model Call Protocol (MCP). Speakers highlight that complex agent interactions are often hampered by chatty, stateful handshakes and reliance on visual/pixel-based inference (the 'guessing game'). Solutions proposed include implementing Web MCP, which allows front ends to expose capabilities rather than just pixels, and building robust guardrails and validation logic directly into the protocol's plumbing (using lifecycle hooks) to prevent agents from reinventing existing components or making unauthorized calls.

Key takeaways

  1. Protocol Overhead is High 2:07

    A stateful MCP handshake is highly chatty. For every 10 million protocol messages, 1.2 million are 'initialize' events, but only 62,000 are actual tool calls, indicating significant protocol overhead (00:02:07).

  2. Web MCP Shifts Focus from Pixels to Capabilities 2:32

    Current web agents operate by observing screenshots, DOM, or accessibility trees, which is inefficient and consumes excessive tokens. Web MCP proposes letting the front end expose defined capabilities instead of relying on pixel-level guessing (00:02:32).

  3. Guardrails Must Live in the Plumbing 4:57

    Since developers cannot control what an LLM agent decides to call, guardrails must be implemented in the protocol's plumbing (e.g., using lifecycle hooks before or after a tool call) to validate outputs, such as ensuring an email is in a client's custom domain (00:04:57).

Watch on YouTube Full article

Pacing the AI frontier, IBM Granite 4.2 & Meta’s Muse assistant thumbnail

· 38:47

Pacing the AI frontier, IBM Granite 4.2 & Meta’s Muse assistant

This episode provides a deep dive into the current state of frontier AI, covering the debate around slowing development (pacing), the technical specifications of IBM's Granite 4.2 models, and Meta's push into personal agents with Muse. Key technical takeaways include the focus on smaller, auditable models, the use of synthetic data for training, and the critical need for robust sandboxing and guardrails for agentic workflows.

Key takeaways

  1. Pacing AI Development 25:29

    Concerns regarding existential risk and the ability of agents to find 'day zero vulnerabilities' have led to calls for slowing AI development. Experts suggest that the focus should be on implementing guardrails for agent-to-agent communication and limiting concurrent agent calls to mitigate economic risks.

  2. IBM Granite 4.2 Release 25:29

    The new Granite models (3B, 8B, 30B) are designed for enterprise use, featuring native step-by-step reasoning and support for agentic workflows like planning and tool calling. The models are available on Apache 2.0 license via Hugging Face.

  3. Meta's Muse Agent

    Muse is Meta's personal AI agent, designed to run on a secure virtual machine (VM) for isolation. While technically advanced in its design, the discussion highlights ongoing concerns regarding user privacy and the security risks of handing over personal data to such agents.

Watch on YouTube Full article

AI Agents Are Starting To Buy. Stripe Is Building How They Pay. thumbnail

· 30:49

AI Agents Are Starting To Buy. Stripe Is Building How They Pay.

The video explores the emerging economic infrastructure required for AI agents to become reliable, autonomous economic actors. The core argument is that the shift from human-driven commerce to agent-driven commerce necessitates fundamental changes in trust, billing, and fraud detection. Stripe, in particular, is building the necessary financial rails, moving the focus from simple transaction-level fraud to pre-transaction abuse, such as stealing tokens. Key architectural shifts include adopting usage-based and outcome-based pricing models, and developing sophisticated systems to manage agent autonomy and financial risk.

Key takeaways

  1. AI agents are fundamentally changing commerce, making the infrastructure the bottleneck. 0:20

    The current AI wave is shifting commerce from simple tasks to complex buying and selling. The challenge is building a system that can be 100% trusted when a computer (agent) is making purchase decisions on behalf of a user. (00:00, 02:00)

  2. Fraud prevention must evolve from transaction-level to pre-transaction abuse detection. 20:30

    Traditional fraud systems (like Radar) focus on the moment of transaction. However, the new threat involves 'stealing tokens' or abusing free credits *before* a transaction occurs, requiring a shift to identifying abusive or risky customers across the entire network. (07:57, 12:30)

  3. The future of billing requires moving beyond subscriptions to outcomes.

    The industry is moving toward outcome-based pricing, where customers pay for the delivered value (the outcome) rather than fixed subscriptions or raw resource costs (like GPU/inference cost). This requires complex billing stacks and machine-readable payment protocols. (03:27, 24:59)

  4. Agent autonomy requires a dedicated 'wallet' and explicit user approval rails.

    For agents to operate safely, a dedicated wallet (like Stripe Link) is needed to manage spending. Crucially, the user must maintain control, requiring explicit approval rails for spending, even if the agent is goal-motivated. (07:33, 32:00)

Watch on YouTube Full article

If we want them to do Knowledge Work, design them as Knowledge Agents — Benjamin Clavié, Mixedbread thumbnail

· 17:55

If we want them to do Knowledge Work, design them as Knowledge Agents — Benjamin Clavié, Mixedbread

The talk distinguishes between 'Coding Agents' and 'Knowledge Agents,' arguing that most real-world tasks fall under the latter. While code provides durable cues (identifiers, file paths), knowledge work is inherently ambiguous, diffuse, and context-dependent (e.g., legal or medical research). The speaker posits that AI agents must be designed to mimic human knowledge work patterns—specifically, through advanced orchestration. This involves breaking down complex, open-ended problems, utilizing multiple specialized tools (primitives like BM25 and semantic search), and employing sub-agents (searchers) to synthesize findings into memos, thereby reducing the 'oracle gap' between perfect knowledge and the agent's output.

Key takeaways

  1. Knowledge Work vs. Coding Work

    Coding is a special, easy case because code has durable cues (identifiers, method definitions). Knowledge work, however, is defined by ambiguous information input and requires reconstructing intent and judgment, making it significantly harder for agents.

  2. The Knowledge Loop 9:15

    Human progress in knowledge work is driven by a single self-optimizing loop: better tools create new roles, new roles generate knowledge, and new knowledge demands better tools. This pattern should guide agent design.

  3. The Necessity of Orchestration

    Effective agent performance requires more than just a single tool. The most significant gains come from architectural improvements, such as having a main agent delegate tasks to specialized 'searcher agents' that return structured memos, which reduces the 'oracle gap' (the difference between perfect information and the system's output) by up to 40%.

  4. Tooling is not Neutral

    Tools are not merely incremental improvements; they are critical for overcoming performance ceilings. The ability to use a tool (e.g., a library catalog vs. physically searching archives) determines if a task is scalable and cheap enough to be practical.

Watch on YouTube Full article

Docker, Adobe & tldraw: Where Should Your Agent Run? thumbnail

· 10:08

Docker, Adobe & tldraw: Where Should Your Agent Run?

The discussion explores the critical architectural question of where AI coding agents should execute, presenting four distinct models: Docker advocates for secure microVM sandboxes; Helix ML proposes centralized, dedicated computing resources for each agent; Adobe demonstrates running the agent loop entirely within the browser tab; and tldraw visualizes agents collaborating as characters on an infinite canvas. The consensus highlights the trade-offs between isolation, centralized control, and environmental fidelity.

Key takeaways

  1. MicroVMs for Agent Sandboxes 0:09

    Docker recommends using micro VMs instead of traditional containers for agent sandboxes because enterprise security teams view shared kernels as an unacceptable isolation boundary.

  2. Centralized Agent Infrastructure 3:03

    Helix ML argues for giving every agent its own dedicated computer on centralized infrastructure (e.g., Kubernetes) to facilitate seamless handoffs of work across global time zones.

  3. Browser-Native Agent Loops 7:33

    Adobe demonstrated an agent that runs its entire loop and controls the browser from within the browser tab, showcasing the concept of the 'self-licking ice cream cone' (SLICC).

  4. Collaborative Canvas Agents 6:13

    tldraw presents agents as interactive characters on a canvas that can coordinate, plan, and execute tasks as a team, allowing for simultaneous, visible collaboration.

Watch on YouTube Full article

The Search Engine for the Agentic Web — Will Bryk, Exa thumbnail

· 17:49

The Search Engine for the Agentic Web — Will Bryk, Exa

The talk introduces Exa, a search engine designed specifically for AI agents, arguing that traditional search engines (like Google) are fundamentally flawed because they are built for human recommendation rather than precise information retrieval. The core premise is that as AI systems become ubiquitous, the volume of searches issued by machines will surpass human searches by a thousandfold by 2026. Exa addresses this by providing a high-quality, customizable search API that enables agents to perform complex, database-like queries over the world's information, moving beyond simple keyword matching.

Key takeaways

  1. AI Search Volume Prediction

    The speaker predicts that in 2026, the number of searches issued by AI systems will exceed those issued by humans. This volume is expected to increase by a thousandfold in the following years, necessitating a specialized search infrastructure.

  2. The Flaw of Recommendation Engines 5:10

    Mainstream search engines are designed as recommendation engines, meaning they prioritize suggesting related content (e.g., 'shirts with stripes' when 'shirts without stripes' is queried) rather than providing exact, database-level answers. AI agents require the opposite: perfect retrieval.

  3. The Perfect Search Thought Experiment 10:05

    The ideal search involves running a Language Model (LLM) over a complex query and a document pair to determine a match. While this is highly accurate, scaling it to a trillion documents costs millions per query, making cost-optimization the primary engineering challenge.

  4. The Business Catalyst 13:45

    The company's business model was catalyzed when external users requested programmatic API access to their search engine, proving the need for an AI-focused search API, rather than just a consumer product.

Watch on YouTube Full article

Tolan: Voice-First AI Companion — Paula Dozsa, Tolan thumbnail

· 15:08

Tolan: Voice-First AI Companion — Paula Dozsa, Tolan

Paula Dozsa, an engineer on the Tolan team, details the unique engineering challenges of building a voice-first AI companion compared to traditional text-based LLM applications. She emphasizes that voice introduces 'conversational volatility' (fast turns, interruptions) which requires fundamental shifts in pipeline design, including smart turn detection, tiered model routing based on emotional stakes, and rebuilding context per turn rather than relying on a continuous cache. The talk also covers using advanced AI agents (like Claude) to build the product itself, achieving significant improvements in stability and feature development.

Key takeaways

  1. Voice vs. Text LLM Assumptions 4:05

    Text chat assumes slow turns and stable context, while voice is characterized by fast turns and volatile context (interruptions, subject changes mid-sentence). This volatility requires building for messy, real-world speech patterns.

  2. Optimizing for Interruptions 5:38

    Instead of minimizing interruptions, the team focused on building smart turn-taking that reads speech patterns to prevent early, incorrect agent interventions. They paid an extra 60 milliseconds of latency to achieve this.

  3. Memory as Retrieval System 12:32

    To handle volatile context, memory is treated as a retrieval system, not a transcript. Facts and preferences are embedded, stored in a vector database (with sub-50ms lookups), and compressed nightly to resolve contradictions and merge duplicates.

  4. Context Reassembly

    Context must be reassembled from parts every single turn (summary, user persona, retrieved memories, tone guidance) because reusing old context is a 'trap' when the user pivots subjects.

Watch on YouTube Full article

Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind thumbnail

· 18:28

Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind

The presentation details the evolution of LLM agents, demonstrating a shift from complex, brittle Python code loops to declarative, file-based definitions using system instructions and skills. The speaker shows that modern agent architectures, such as the Gemini API's anti-gravity agent, utilize a hosted sandbox and network proxy to manage state and credentials securely. This allows agents to operate using general-purpose tools (like GitHub CLI or Google Search) defined in files (e.g., `AGENTS.md`), drastically reducing the need for thousands of lines of custom orchestration code.

Key takeaways

  1. The Agent Evolution: Code to Files

    Agent development is moving away from writing explicit Python loops, JSON schemas, and tool routing logic. The core functionality is now expressed in files (Markdown/Skills) that define instructions, rules, and capabilities, allowing the model to use general tools.

  2. Server-Side State Management 14:02

    The new architecture handles complex tasks by moving loops, tool routing, session state, and context compaction to the server side, requiring only a single API call with new inputs.

  3. Security and Isolation 12:32

    A hosted sandbox and network proxy ensure that the agent never sees the actual credentials, injecting tokens only when outbound requests are made, and allowing domain restriction for enhanced security.

  4. Focus on Domain Logic 16:40

    The primary work for developers is now defining the domain instructions, rules, and evaluation criteria (the 'what'), rather than writing the infrastructure code (the 'how').

Watch on YouTube Full article

How We Solved Agent Building — Andrew Qu, Vercel thumbnail

· 17:34

How We Solved Agent Building — Andrew Qu, Vercel

Andrew Qu details Vercel's journey in solving agent building, evolving from initial manual data querying to complex, multi-step agent architectures. The major breakthrough was realizing that the core capability was not a clever toolset, but the simple file system (list, read, bash). This insight led to the development of Eve, a framework designed to bring file system conventions (like Next.js) to agents, enabling developers to build highly customized, production-ready agents with minimal effort.

Key takeaways

  1. The File System Unlock 11:57

    The most significant breakthrough was realizing that powerful agents could be built using a minimal set of tools—specifically, file system operations (list, read, run bash)—which models are already well-trained on. This approach allowed agents to explore and write where needed, leading to a massive increase in performance (the eval score basically doubled).

  2. Focus on Company-Specific Knowledge 17:00

    While off-the-shelf agents are useful, the speaker argues that the most effective agents are those stuffed with deep, company-specific knowledge. These custom agents provide the most 'juice' and save significant time across various departments (e.g., legal, finance, data science).

  3. The Need for a Framework (Eve) 15:40

    To simplify agent development, Vercel built Eve, described as the 'Next.js for agents.' It adopts file system conventions, allowing developers to easily declare skills, tools, and channels, making the process of building and iterating on agents highly manageable.

Watch on YouTube Full article

No Memory, No Harness: Why the Database Is the Last Line of Defense — Kay Malcolm, Oracle thumbnail

· 21:37

No Memory, No Harness: Why the Database Is the Last Line of Defense — Kay Malcolm, Oracle

Kay Malcolm argues that while AI models (agents) are powerful, they are incomplete without a robust, centralized memory system. She frames the agent as the 'brain,' the surrounding system as the 'harness' (body), and the database as the 'central nervous system' (memory). The core problem addressed is that current systems (like Git) track code changes, not the human intent or context behind them. To solve this, she advocates for using a unified Oracle AI database to store all five types of agent memory (short-term, long-term, episodic, procedural, and semantic) in a single source of truth, preventing context loss when scaling to enterprise teams.

Key takeaways

  1. AI's Limitation: Code vs. Intent 5:36

    AI tools make individuals faster, but they do not automatically make teams more productive if the context and reasoning behind the code are not shared. Git only records what changed, not the human intent (3:36).

  2. The Agent Architecture Model 11:54

    An enterprise agent requires three components: the Model (the brain), the Harness (the body, enabling action), and Memory (the central nervous system, carrying context) (7:14).

  3. The Five Types of Memory 13:29

    Effective agent memory must distinguish between: short-term (within a session), long-term (across sessions), episodic (what happened last time), procedural (steps taken), and semantic (meaning) (8:09).

  4. The Need for a Single Source of Truth 17:34

    When data is spread across multiple specialized databases (relational, document, graph, vector), agents struggle to reconcile the truth, often guessing incorrectly and wasting tokens (10:54).

  5. The Solution: Unified Database Memory

    A unified database (like the Oracle AI database) is necessary to store all memory types (JSON, relational, graph, vector) in one place, ensuring the agent's memory is non-negotiable and accessible across the entire team (14:24).

Watch on YouTube Full article

We let an AI agent execute Bash and lived to talk about it — Sarah Sanders, PostHog thumbnail

· 21:00

We let an AI agent execute Bash and lived to talk about it — Sarah Sanders, PostHog

The talk details the architecture and security challenges of building agentic CLI tools, exemplified by PostHog's 'Wizard.' The speaker emphasizes that an agent with command execution capabilities is essentially a 'malware starter pack.' Security must be layered, deterministic, and must account for dangerous inputs originating not only from users but also from the system's own supply chain (the 'Context Mill'). The solution involves implementing a dedicated, deterministic scanner (Warlock) and ensuring that the LLM layer acts only as an 'adviser,' never an enforcer.

Key takeaways

  1. The Danger of Agentic Tools 0:03

    An agent that can run commands is structurally similar to malware, requiring extreme caution. The threat model must account for how 'attacks compose' from multiple, seemingly innocent vulnerabilities.

  2. Context Supply Chain Risk 0:11

    The most dangerous input can come from the system's own supply chain (the Context Mill), such as a seemingly harmless prompt injection payload signed by the development team.

  3. Security Must Be Deterministic 0:15

    For enforcement, security mechanisms must be deterministic and mechanical. The LLM layer should only be an 'adviser' to reduce noise, never the primary enforcer, to prevent failure modes.

Watch on YouTube Full article