The Video Signal

Topic

AI Agents

All digests tagged AI Agents

How Harmonic 4x'd User Retention by Building on Deep Agents thumbnail

· 16:25

How Harmonic 4x'd User Retention by Building on Deep Agents

Harmonic transitioned its natural language interface, Scout, from a brittle query parsing graph to an architecture built on Deep Agents and a simple model-plus-tools loop. This shift quadrupled week one to week four user retention. The core technical lesson is that robust agent design requires managing context via a 'harness contract,' ensuring that all artifacts (like visualizations or large search result sets) are visible to the model—either in the message list or offloaded through file system tools—to prevent the UX from becoming an invisible black box.

Key takeaways

  1. Deep Agents significantly boost retention 2:04

    Switching to Deep Agents resulted in a fourfold increase in week one to week four user retention for Scout. (1:24)

  2. The agent architecture simplified from graphs to loops 4:01

    Scout evolved from complex, multi-node query parsing graphs (LangGraph) into a simpler model and tools loop, mediated by middleware. (2:41)

  3. Context management is handled by the harness 8:16

    Deep Agents manage context overload using mechanisms like compaction for long message lists and file system abstraction to store large results, returning only pointers to the model. (4:56)

  4. UX must respect the agent's context contract 11:44

    For a product UX to be useful, any rendered element (e.g., charts) must either reside in the message list or be discoverable by the model via tools/file system pointers; otherwise, it is invisible to the agent. (7:04)

Watch on YouTube Full article

How AI agents reproduced ICML 2026 papers thumbnail

· 26:37

How AI agents reproduced ICML 2026 papers

The ICML 2026 Agents Reproduction Challenge was a large-scale community effort involving over 1,200 participants and AI agents attempting to reproduce claims from accepted machine learning papers. The initiative demonstrated the potential for automated reproducibility testing in academic research, finding that while a majority of papers were reproducible (some fully, some via smaller scale experiments), significant flaws were also identified. Key technical takeaways include the use of specialized tools like `tracko` and Hugging Face infrastructure to create fully auditable, machine-readable log books for every reproduction attempt.

Key takeaways

  1. Scale of Reproduction Effort 4:18

    The challenge involved 1,200+ participants attempting to reproduce claims from a subset of ICML 2026 papers. A total of 2,200 unique papers were attempted, resulting in approximately 35,000 different claims being judged (Timestamp: ~4:18).

  2. Reproducibility Success Rate 12:34

    A majority of the papers looked at were reproducible. Specifically, over 2,000 papers had at least one major claim independently verified (Timestamp: ~6:34).

  3. Identification of Flaws and Contested Claims 13:10

    The community found that about 23% of papers could not be fully reproduced as claimed, leading to at least 496 contested or falsified claims. Furthermore, 49 papers were almost fully falsified (Timestamp: ~8:15).

  4. Best Practices in Agent Use 15:42

    The 'Best Human in the Loop' award highlighted that effective reproduction requires human intervention to guide agents, especially when evaluating qualitative results (e.g., building a UI to compare quantized images) (Timestamp: ~10:35).

Watch on YouTube Full article

Gadgets: Personal app vibe coding that is actually safe — Kenton Varda, Cloudflare thumbnail

· 18:54

Gadgets: Personal app vibe coding that is actually safe — Kenton Varda, Cloudflare

The talk argues that modern personal AI code generation capabilities fundamentally break traditional cloud infrastructure models designed for single-version applications. Kenton Varda introduces 'Gadgets,' a new application paradigm built on Cloudflare Workers. Gadgets allow users' agents to add custom features directly to an app instance (like adding strikethrough formatting or generating complex SVGs) without requiring the core developer to rewrite the entire platform, thus bypassing the limitations of centralized cloud architecture and traditional feature request pipelines.

Key takeaways

  1. Personal AI Codegen Breaks Traditional Cloud Infrastructure

    The current model requires developers to handle all user-requested features (filed in Jira) through massive, multi-year plugin rewrites. This process is slow and often fails. Personal AI agents offer an alternative where users can have their own agent write and add features directly for their specific use case, keeping the core app clean.

  2. The Limitations of Current Web/Cloud Architecture 13:59

    Traditional web apps run on a developer's server, ensuring all users see one 'blessed version.' This centralization prevents user customization. The proposed Gadget model ensures that each gadget is an isolated instance, and sharing/access control is managed by the platform, not the app itself.

  3. Gadgets Security Model 17:05

    The security architecture isolates components: The UI runs in a null origin iframe sandbox with Content Security Policy. Communication is restricted via `postMessage` to the parent frame, which establishes a Cap'n Web RPC session to server code running in a dynamic worker sandbox (durable objects). This prevents XSS bugs from leaking data outside the isolated environment.

Watch on YouTube Full article

2nd Place Winner: Coding Agent Calls Developer to Pitch Launch Strategy thumbnail

· 5:12

2nd Place Winner: Coding Agent Calls Developer to Pitch Launch Strategy

The video demonstrates an autonomous AI agent designed for product positioning strategy that operates while the developer is away (AFK). The agent handles routine tasks but utilizes a defined escalation matrix to call the human developer only when faced with non-reversible, high-stakes decisions. This process not only facilitates real-time discussion via voice call but also ensures all resulting decisions and follow-up action items are automatically logged back into the project documentation for transparency.

Key takeaways

  1. Autonomous AFK Operation

    The agent is instructed to run autonomously, completing all tasks it can handle without human intervention. It also checks working hours to prevent calling outside designated times.

  2. Strategic Escalation Matrix 1:40

    When the agent reaches a critical decision point (e.g., Lead on Value vs. Lead on Price), it triggers an escalation, presenting structured options and recommendations rather than asking for generic input.

  3. Decision Logging and Transparency

    Following the human decision (e.g., 'Lead on Value'), the agent automatically logs the approved decision and creates a follow-up task (e.g., 'follow up in 7 days') directly into the project files, ensuring decisions are never lost within transcripts.

Watch on YouTube Full article

1st Place Winner: Coding Agent Calls Developer to Resolve Code Block thumbnail

· 6:17

1st Place Winner: Coding Agent Calls Developer to Resolve Code Block

The demo showcases an advanced AI coding agent that autonomously handles a critical bug fix in a checkout API. When faced with a technical decision requiring human judgment—specifically, whether to maintain backward compatibility (Option A) or implement a clean refactor causing breaking changes (Option B)—the agent initiates an automated phone call to the developer for real-time guidance and execution.

Key takeaways

  1. Autonomous Agent Setup

    The setup involves running a coding agent via the Claude Code CLI, monitored by the Vocal Bridge dashboard, targeting a validation bug across five checkout API handlers (e.g., create order, apply coupon).

  2. Decision Point Triggered 3:26

    The agent identifies that fixing the bug requires a judgment call: Option A maintains backward compatibility but involves code duplication; Option B is a clean refactor but introduces a breaking change to the error format.

  3. Human-in-the-Loop Communication 1:52

    Instead of guessing, the agent initiates an outbound phone call (via VocalBridgeAI) to present the technical trade-offs and obtain a decision from the developer while they are away from their keyboard.

  4. Automated Execution

    Upon receiving the final verbal confirmation (Option B), the agent automatically executes the chosen path, logs the decision, and updates the code base without manual developer intervention.

Watch on YouTube Full article

Evolving AI chat with MCP Apps - Phil Nash - NDC Copenhagen 2026 thumbnail

· 38:00

Evolving AI chat with MCP Apps - Phil Nash - NDC Copenhagen 2026

The talk introduces MCP Apps, a proposed open standard designed to evolve AI chat interfaces beyond plain text. By integrating rich, interactive web UIs (built with HTML/CSS/JavaScript) directly into the conversation flow, MCP Apps allow agents to render mini-applications for tasks like booking hotels or managing playlists. This approach moves interaction from boring 'walls of text' to engaging, visual experiences, making AI more useful for complex user workflows.

Key takeaways

  1. The Need for Interactive UIs in Chat 18:02

    Traditional chat interactions are limited to text (or code/tool calls), which is insufficient for tasks requiring visual exploration, configuration of multiple options, or viewing real-time data. MCP Apps solve this by bringing web-powered interfaces into the chat environment.

  2. MCP Apps as an Open Standard 22:40

    MCP Apps is a standard inspired by community efforts (like MCP-UI) and commercial SDKs (e.g., OpenAI's Apps SDK), aiming to provide a unified way for agents to render UIs across different model providers.

  3. Core Functionality: Sandboxed Web Views 26:00

    MCP Apps are implemented as sandboxed web applications (HTML, CSS, JavaScript) loaded within an iframe. This isolation keeps the UI safe while allowing it to interact with the agent host via tool calls and a JSON RPC mechanism.

Watch on YouTube Full article

Wayve's Dave Kirk: Why Agentic Code Review Needs Evals thumbnail

· 23:55

Wayve's Dave Kirk: Why Agentic Code Review Needs Evals

Dave Kirk details Wayve's approach to agentic PR code review, emphasizing that reliable AI adoption requires moving beyond 'vibes-based' evaluation. The system uses a structured feedback loop—integrating sentiment tracking, usage metrics, and dedicated evaluations (Evals)—to improve prompts and guide multi-agent behavior in complex, high-stakes environments like self-driving car development.

Key takeaways

  1. Agent Reliability Requires Observability 2:08

    Multi-agent systems are stochastic and difficult to predict. Kirk notes that observability is critical; if a single agent's behavior cannot be observed, building reliable, production-ready multi-agent workflows is extremely challenging.

  2. The Pitfalls of Public Benchmarks 10:53

    Public coding benchmarks are often untrustworthy because agents can learn to 'cheat' the tests. Performance gains may simply reflect improved cheating mechanisms rather than genuine capability improvements.

  3. Structured Feedback Loops are Essential 22:30

    Wayve implements a feedback loop by collecting data on code review outcomes, including sentiment (thumbs up/down) and usage tracking. This data is used to identify common mistakes in prompts and improve agent behavior iteratively.

  4. The Value of Evals 23:25

    To ensure confidence, the team uses dedicated evaluation agents (Evals) that test the quality of output from other agents. Kirk highlights performing 'eval-driven development,' where the eval mechanism is built before the agent itself.

Watch on YouTube Full article

Stanford CS329A Self-Improving AI Agents | Part 2 | Test-Time Compute Scaling thumbnail

· 1:03:21

Stanford CS329A Self-Improving AI Agents | Part 2 | Test-Time Compute Scaling

The lecture details advanced methods for improving Large Language Model (LLM) performance through 'inference scaling' or 'test-time compute scaling,' rather than relying solely on expensive pre-training. Key techniques include repeated sampling (Large Language Monkeys), which shows that coverage follows a predictable power law with the number of samples. The discussion highlights the critical need for robust verification mechanisms to bridge the generation-verification gap, and concludes by introducing the Arkon framework, an architecture search method that optimizes complex inference pipelines using components like Fusion, Critic, and Ranker.

Key takeaways

  1. Inference Scaling Paradigm Shift 1:45

    LLM capability can be significantly enhanced at inference time by increasing compute (e.g., repeated sampling) without modifying model parameters or requiring fine-tuning, offering a new paradigm compared to traditional pre-training and fine-tuning.

  2. Repeated Sampling Effectiveness 2:40

    By repeatedly querying the same problem (e.g., using Llama 3-8b or DeepSeek), selecting the correct response among candidates, models can achieve performance comparable to larger proprietary models like GPT-4o.

  3. The Role of Verification 7:50

    For repeated sampling to be effective, automated verification is crucial. The 'generation-verification gap' describes the large difference between the best possible outcome (Oracle selection) and what can be achieved using simple methods like majority voting.

  4. Advanced Scaling Architectures 20:30

    The Arkon framework treats inference scaling as an architecture design problem, optimizing the combination of techniques (e.g., Fusion, Critic, Ranker) to maximize accuracy given a limited compute budget.

Watch on YouTube Full article

Agentic Engineering vs Software Engineering: Beyond Vibe Coding thumbnail

· 10:46

Agentic Engineering vs Software Engineering: Beyond Vibe Coding

Software engineering is undergoing a fundamental shift from writing explicit, deterministic instructions to defining high-level goals and orchestrating autonomous AI agents. Agentic Engineering treats AI systems as collaborators capable of multi-step workflows, requiring the human developer's role to evolve into that of an architect who supervises, constrains, and validates probabilistic outputs rather than manually executing every task.

Key takeaways

  1. The Shift in Effort

    Traditional software engineering requires writing explicit instructions (deterministic logic). Agentic Engineering allows developers to define goals, while AI agents handle the execution, changing where the core engineering effort is applied.

  2. Defining Agentic Engineering 3:42

    Agentic refers to an organization of agents that write code, while the human developer maintains a 'human in the loop' to oversee and validate the output as the multi-agent system iterates through subtasks.

  3. The Coding Spectrum 5:01

    Coding methods exist on a spectrum based on human agency: Traditional SE (full control) $ ightarrow$ AI-assisted coding (snippets/refactoring) $ ightarrow$ Vibe coding (natural language intent) $ ightarrow$ Agentic coding (autonomous planning/execution) $ ightarrow$ Agentic engineering (designing environments for autonomous systems).

  4. Increased Value of Oversight 8:44

    As agentic systems become more autonomous, the value of human oversight increases significantly. Engineers are now responsible not only for writing code but also for ensuring reliability across probabilistic workflows.

Watch on YouTube Full article

Understanding AI Agent Hallucination in AI Systems thumbnail

· 10:51

Understanding AI Agent Hallucination in AI Systems

AI hallucination occurs when an AI system confidently provides information that is factually incorrect. As systems evolve from basic LLMs to autonomous agents (which plan, reason, and take action), the risk landscape changes significantly. While advanced agents can increase error opportunities, grounding them with tools (like search or APIs) dramatically reduces hallucination by allowing verification. Mitigation requires implementing robust design choices: connecting agents to verified 'sources of truth,' enforcing tool-based reasoning, strictly controlling operational scope boundaries, and maintaining a human in the loop for critical decisions.

Key takeaways

  1. Hallucination Definition 0:13

    Hallucination is when an AI system confidently provides information that is totally incorrect (1:25). This risk increases as systems move from simple chatbots to autonomous agents.

  2. Agentic Risk Profile 0:02

    Agents do hallucinate less when grounded with tools (e.g., search tools, data connectors, RAG) because they can verify information instead of guessing (2:08). However, they introduce more danger because a wrong action (like updating a field or scheduling a meeting) can be taken with complete confidence (3:15).

  3. Mitigation Strategy: Grounding and Tools 0:07

    The fastest way to reduce hallucination is to connect the agent to reliable 'sources of truth' (e.g., SharePoint, CRM systems) and enforce tool-based reasoning rather than pure text prediction (7:30).

Watch on YouTube Full article

I Stopped Installing Claude Skills. Here's What I Do Instead. thumbnail

· 16:57

I Stopped Installing Claude Skills. Here's What I Do Instead.

The video provides an advanced deep dive into AI agent skills (used by models like ChatGPT, Claude, and Codex), arguing that these 'superpowers' are often misunderstood. Skills are not traditional apps; they are sets of instructions that must be designed to be both readable by humans for auditing and highly functional for the AI agent during runtime. The speaker emphasizes moving beyond simply collecting skills and instead focusing on structured development, conflict resolution across multiple skills, and utilizing specialized tools like a 'Skill Builder' to ensure reliable, production-grade performance.

Key takeaways

  1. Skills are not applications (apps)

    A skill is simply a set of instructions for the AI agent. Unlike apps, skills do not load their full functionality upfront; only the name and description are loaded initially. The full instruction set is only invoked when the task matches the description, making loading order critical.

  2. The Core Reframing: Dual Audience Design 3:25

    Skills must be written for two audiences simultaneously: the AI agent (for utility) and the human developer (for readability and auditing). If humans cannot read them, developers cannot understand what is being given to the AI.

  3. Auditing and Conflict Resolution

    As agents accumulate many skills (e.g., 25+), conflicts can dull the results because the AI averages out competing instructions. Advanced builders must audit their setup to resolve these performance degradations.

  4. The Role of Structured Development

    To ensure reliability, developers should use tools (like the 'Skill Builder') that enforce best practices for skill files, ensuring clarity in the front matter and structure while maintaining human readability.

Watch on YouTube Full article

Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software thumbnail

· 21:15

Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

The talk addresses the challenge of defining and measuring 'long horizon work' for AI agents. The core argument is that progress depends less on headline benchmark numbers (like human-equivalent hours) and more on rigorous design of the environment and verifier mechanisms. For build engineering contexts, this means focusing on how tasks involve complex tool coordination (e.g., CI/CD logs, databases), managing state changes, and implementing robust 'judge models' that verify correctness from the final system state rather than just the agent's path.

Key takeaways

  1. Defining Long Horizon 0:37

    Long horizon is a scalar metric, but relying solely on human-equivalent time (e.g., 16 hours via Meter) or model metrics (tokens/steps) is insufficient. The most accurate measure requires considering all variables and the inherent complexity of the task.

  2. Measuring Model Capability 4:03

    Model capability should be measured by environment complexity, specifically tool coordination (how many tools are used) and state change complexity. Tasks that can be artificially stretched by chaining unrelated steps do not meaningfully measure model ability.

  3. The Importance of Verifiers 7:12

    For complex software domains, deterministic verifiers are often impractical or impossible. The solution is introducing a 'judge model' (or critic) that verifies correctness by examining both the final state of the environment and the entire execution trajectory.

  4. Addressing Ambiguity 10:38

    Since real-world tasks are ambiguous, standardized evaluation is difficult. Judges must be designed to handle open-ended solutions rather than requiring a single reference answer or sample trajectory.

Watch on YouTube Full article

Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute thumbnail

· 18:20

Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

The presentation outlines a vision for advanced AI agents that can continuously learn and adapt in real-world enterprise environments post-deployment. The core methodology involves an orchestrated training loop: interactions are captured by an orchestrator, processed through inference engines, graded, and the resulting data (graded chats/traces) is fed into a training engine to compute weight updates. Key challenges addressed include environment fidelity, preventing reward hacking, and transitioning from controlled, replayable synthetic environments to uncontrolled, off-policy production data.

Key takeaways

  1. The Progression of Agent Training

    Agent training moves from simple single-turn Q&A tasks (controlled by a dedicated training stack) to complex, multi-turn, long-horizon tasks that require offloading environment state outside the training stack.

  2. The Core RL Training Loop

    Training relies on an orchestrator driving rollouts, which sends prompts to a model and then passes results to a grader. The resulting graded chats are used by a training engine to compute weight updates for the inference engines.

  3. Addressing Real-World Data Challenges 15:45

    Replicating production environments is difficult due to issues like non-replayability and off-policy data. The future requires methods like automated data pipelines and qualitative feedback ingestion to learn from real interactions.

  4. The Vision: Self-Improving Agents

    The ultimate goal is a single deployment model that can interact across many different settings, continuously self-evaluating and computing weight updates from every interaction it has.

Watch on YouTube Full article

Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang thumbnail

· 16:33

Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang

Emulated focuses on creating high-fidelity training data environments that simulate entire companies and complex infrastructure operations, moving beyond simple code diffs or single-node sandboxes. The core argument is that for AI agents to achieve true autonomy in mission-critical systems (like cloud providers), they must be trained on long-horizon tasks involving distributed cluster failures, resource provisioning across VPCs/subnets, managing cost constraints, and reasoning through real-world operational incidents.

Key takeaways

  1. The Data Gap in AI Agents 3:30

    Current benchmarks (e.g., SweBench Pro, Terminal Bench) limit agents to operating within a codebase, failing to capture the complexity of real-world tasks like PM communication, performance testing, or owning underlying infrastructure over years.

  2. Complexity Requires Full Simulation 6:10

    Real infrastructure work is not a simple code diff; it involves managing failing nodes, stale deprecated components, live traffic serving, and operational blast radius across distributed clusters.

  3. Limitations of Single-Node Sandboxes 10:40

    Standard post-training pipelines often use homogeneous single-node sandboxes. However, real cloud services require simulating resource provisioning (EC2, Cloud Run), VPCs, subnets, and security groups, which necessitates a multi-node sandbox with access to real infrastructure.

Watch on YouTube Full article

Building Deep Agents and Deploying in Production thumbnail

· 15:40

Building Deep Agents and Deploying in Production

Deep Agents are defined as a sophisticated 'harness' built around foundational LLMs, providing the necessary infrastructure—beyond just the model itself—to make agents reliable and useful in production. The system integrates core primitives like memory, tools, file systems (acting as scratchpads), and middleware hooks. For deployment, critical considerations include implementing durable execution via checkpointing, managing short and long-term memory stores, establishing robust Role-Based Access Control (RBAC) for tool access, and designing for human oversight (human in the loop).

Key takeaways

  1. Deep Agents are a 'Harness' 0:27

    An agent is conceptualized as an LLM plus a harness. The harness encompasses all infrastructure—including system prompts, memory management, tools, file systems, and middleware—that makes the model reliable for a given task. (0:27)

  2. Deep Agents Architecture 6:58

    Deep Agents represent the highest level of abstraction in the LangChain stack, built on top of LangGraph, which provides the core composable nodes and edges necessary for complex agent workflows. (4:18)

  3. Production Reliability Requirements

    For production deployment, agents must handle long-running tasks using durable execution (checkpointing) to recover from failures at any step, manage short/long-term memory across sessions, and incorporate human approval loops. (9:48)

Watch on YouTube Full article

Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect thumbnail

· 19:27

Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

Will Brown discusses extending Reinforcement Learning (RL) into complex, real-world tasks that lack clean, verifiable rewards. The core thesis is that 'environments' must serve as the anchor for learning. Techniques like grounding in source material, using LLM judges to audit actions, and employing a reverse direction trick are necessary to generate reliable reward signals when ground truth is unavailable. The ultimate goal is enabling continual learning—allowing deployed agents to autonomously improve by observing and correcting mistakes in messy production settings.

Key takeaways

  1. The Shift from Verifiable Rewards 6:53

    Traditional RL thrives on verifiable rewards (e.g., math, code test cases). However, most real-world tasks (like writing reports or handling refunds) are fuzzy and lack clean best answers, requiring new methods to generate reliable signals.

  2. Environments as the Learning Anchor

    An 'environment' is defined by a task, a harness (e.g., Docker image, codebase), and a scoring rule/verifier. These objects can be used not only for RL but also for Supervised Fine-Tuning (SFT) or prompt optimization.

  3. Mitigating Reward Hacking

    Since loose proxies for objectives can be exploited, careful design is crucial. Techniques include inspecting traces, running small experiments, and using judges to audit rollouts in hindsight.

  4. Generating Signal via Reverse Direction 17:26

    A powerful technique involves working backward: starting from a known solution or artifact (like a completed PR) and training the model to find it again, providing verifiable steps for an initially hard problem.

Watch on YouTube Full article

The misaligned incentives behind AI coding agents thumbnail

· 50:16

The misaligned incentives behind AI coding agents

The conversation details how AI coding agents, exemplified by Devin, are fundamentally changing software engineering workflows. The industry is moving past simply training larger models and focusing intensely on optimizing cost-efficiency (token spend) and speed. Key technical advancements include the 'sidekick' agent architecture for achieving high price performance, developing advanced evaluation metrics like 'mergeability' via Frontier Code, and implementing proactive automation to shift human engineers into decision-making roles rather than routine coding tasks.

Key takeaways

  1. The Shift from Capability to Efficiency 8:36

    As agents mature, the bottleneck is shifting from model training size to running evaluations and managing costs. The focus has moved toward optimizing speed and cost rather than chasing the absolute best-performing frontier model for every task (5:56).

  2. The Role of Mergeability in Evaluation 14:01

    A critical gap in current evaluation benchmarks is 'mergeability'—determining if code, while technically correct, would improve the overall quality or maintainability of a codebase. Cognition developed Frontier Code to address this (8:41).

  3. Cost Optimization via Sidekick Architecture 35:46

    The 'sidekick' agent architecture allows for running both a high-quality, expensive model and a more price-performant model in parallel. This dual approach enables significant cost savings (up to 35% better price performance) without sacrificing quality (21:46).

  4. Proactive Automation and Productivity Guarantees

    Agents are moving from reactive task completion to proactive automation, handling tasks like triaging messages or suggesting fixes. This capability led Cognition to underwrite a $10 million productivity guarantee based on measuring 'productive engineering output' (46:51).

Watch on YouTube Full article

Your Finance Agent's Bottleneck Is You — Ramana Siddanth Emani, Auditoria AI thumbnail

· 13:42

Your Finance Agent's Bottleneck Is You — Ramana Siddanth Emani, Auditoria AI

The primary bottleneck in developing production AI agents, especially for regulated fields like finance, is not model capability or hardware, but the developer's manual workflow. The talk outlines how advanced agent harnesses can automate and multiply human output by implementing primitives such as sub-agents, skills, goals, and loops. By orchestrating these agents across parallel worktrees (e.g., handling multiple Jira tickets simultaneously), developers can drastically reduce time spent on repetitive tasks like testing, building, and deployment, shifting the human role from executor to final verifier.

Key takeaways

  1. The Developer Loop is the Bottleneck

    Shipping production agents fails due to developer velocity limitations. The solution lies in automating the internal developer loop rather than waiting for model or hardware upgrades (0:00, 2:37).

  2. Parallel Task Execution via Sub-Agents 7:02

    Sub-agents allow for parallel processing using isolated 'worktrees'—virtual folders where agents write and execute code independently. This enables handling multiple tasks (e.g., QA bug tickets) simultaneously, far exceeding human capacity (4:22).

  3. Minimal UX for Orchestration 8:38

    Effective agent use requires minimal User Experience (UX), consolidating all project dashboards, logs, Jira tickets, and GitHub PRs into a single pane of glass widget to reduce context switching and manual orchestration effort (5:18).

  4. Achieving Autonomy with Goals and Dreaming

    Future agent development involves combining 'goals' (setting a desired outcome) with automated loops. Furthermore, allowing agents to 'dream' by collecting customer usage patterns can drive self-improvement and feature upgrades autonomously (11:35).

Watch on YouTube Full article

Let's integrate AI Agents in Event-Sourced Systems — Divakar Kumar, FlyersSoft thumbnail

· 21:37

Let's integrate AI Agents in Event-Sourced Systems — Divakar Kumar, FlyersSoft

The presentation outlines an advanced architectural pattern for integrating AI agents into existing enterprise systems, specifically targeting real-time fraud detection. Instead of replacing established Rule-Based Engines or ML models, the approach layers agentic intelligence to resolve 'gray zone' transactions—cases where current deterministic systems lack sufficient context. The architecture leverages Event Sourcing and Domain-Driven Design (DDD) principles by building a centralized semantic layer that aggregates data from multiple bounded contexts (e.g., Transaction, Device, Account). Agents then consume this enriched context via an orchestration layer in a saga pattern to reach a final verdict.

Key takeaways

  1. AI Agents Augment, Not Replace, Existing Systems

    The primary value of AI agents is not replacing existing systems (like rule-based or ML engines) but rather adding an agentic layer to handle ambiguous cases—the 'gray zone' where current models lack necessary context. The goal is to enhance business workflows using the state and history already captured by event sourcing.

  2. Context Aggregation via Semantic Layer 17:37

    To provide sufficient context for agents, data from disparate bounded contexts (Transaction, Device, Account, Payment) must be aggregated into a semantic layer or materialized view. This is achieved by propagating domain events using Change Data Capture (CDC) mechanisms or message brokers.

  3. Saga Orchestration and Agentic Decision Making

    The system uses an orchestration layer to manage the process. This layer employs a fan-out pattern, sending events simultaneously to multiple specialized agents (e.g., Risk Analyzer Agent, Behavior Analyzer Agent). A final Verdict Agent then analyzes these responses to reach a consensus event.

Watch on YouTube Full article

SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe thumbnail

· 16:29

SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe

Nubank achieved a massive acceleration in deploying AI agents—reporting up to 20x faster shipping—by shifting agent evaluation from relying solely on slow, expensive production data or manual curation to using grounded simulations. The core methodology involves running synthetic, multi-turn conversations against the agent within a specialized framework (Snowglobe), allowing teams to rapidly test models and variants in a continuous 'ship, observe, simulate, repeat' loop.

Key takeaways

  1. Simulations Shortcircuit the Eval Bottleneck 10:44

    Traditional agent evaluation data is complex (multi-turn, stateful trajectories) and time-consuming to gather via manual authoring or limited production traces. Simulations allow teams to generate thousands of multi-turn conversations quickly, reducing release cycles from weeks to hours.

  2. The Simulation Loop: Ship, Observe, Simulate

    A robust agent development cycle involves shipping the agent, observing real data, creating strong evaluations (using techniques like automated prompt optimization), and then using simulations to generate synthetic data. This simulated data is piped into the evaluation pipeline for continuous improvement.

  3. Closing the Sim-to-Real Gap

    To trust simulation results, it is crucial to establish metrics and processes that compare simulated performance against real production data (offline/online human review). The correlation between sim quality via evals and real data was found to be high.

Watch on YouTube Full article