Topic

AI Agents

All digests tagged AI Agents

Jul 30, 2026 · 50:16

The misaligned incentives behind AI coding agents

The conversation details how AI coding agents, exemplified by Devin, are fundamentally changing software engineering workflows. The industry is moving past simply training larger models and focusing intensely on optimizing cost-efficiency (token spend) and speed. Key technical advancements include the 'sidekick' agent architecture for achieving high price performance, developing advanced evaluation metrics like 'mergeability' via Frontier Code, and implementing proactive automation to shift human engineers into decision-making roles rather than routine coding tasks.

Key takeaways

The Shift from Capability to Efficiency 8:36
As agents mature, the bottleneck is shifting from model training size to running evaluations and managing costs. The focus has moved toward optimizing speed and cost rather than chasing the absolute best-performing frontier model for every task (5:56).
The Role of Mergeability in Evaluation 14:01
A critical gap in current evaluation benchmarks is 'mergeability'—determining if code, while technically correct, would improve the overall quality or maintainability of a codebase. Cognition developed Frontier Code to address this (8:41).
Cost Optimization via Sidekick Architecture 35:46
The 'sidekick' agent architecture allows for running both a high-quality, expensive model and a more price-performant model in parallel. This dual approach enables significant cost savings (up to 35% better price performance) without sacrificing quality (21:46).
Proactive Automation and Productivity Guarantees
Agents are moving from reactive task completion to proactive automation, handling tasks like triaging messages or suggesting fixes. This capability led Cognition to underwrite a $10 million productivity guarantee based on measuring 'productive engineering output' (46:51).

Watch on YouTube Full article

Jul 30, 2026 · 13:42

Your Finance Agent's Bottleneck Is You — Ramana Siddanth Emani, Auditoria AI

The primary bottleneck in developing production AI agents, especially for regulated fields like finance, is not model capability or hardware, but the developer's manual workflow. The talk outlines how advanced agent harnesses can automate and multiply human output by implementing primitives such as sub-agents, skills, goals, and loops. By orchestrating these agents across parallel worktrees (e.g., handling multiple Jira tickets simultaneously), developers can drastically reduce time spent on repetitive tasks like testing, building, and deployment, shifting the human role from executor to final verifier.

Key takeaways

The Developer Loop is the Bottleneck
Shipping production agents fails due to developer velocity limitations. The solution lies in automating the internal developer loop rather than waiting for model or hardware upgrades (0:00, 2:37).
Parallel Task Execution via Sub-Agents 7:02
Sub-agents allow for parallel processing using isolated 'worktrees'—virtual folders where agents write and execute code independently. This enables handling multiple tasks (e.g., QA bug tickets) simultaneously, far exceeding human capacity (4:22).
Minimal UX for Orchestration 8:38
Effective agent use requires minimal User Experience (UX), consolidating all project dashboards, logs, Jira tickets, and GitHub PRs into a single pane of glass widget to reduce context switching and manual orchestration effort (5:18).
Achieving Autonomy with Goals and Dreaming
Future agent development involves combining 'goals' (setting a desired outcome) with automated loops. Furthermore, allowing agents to 'dream' by collecting customer usage patterns can drive self-improvement and feature upgrades autonomously (11:35).

Watch on YouTube Full article

Jul 30, 2026 · 21:37

Let's integrate AI Agents in Event-Sourced Systems — Divakar Kumar, FlyersSoft

The presentation outlines an advanced architectural pattern for integrating AI agents into existing enterprise systems, specifically targeting real-time fraud detection. Instead of replacing established Rule-Based Engines or ML models, the approach layers agentic intelligence to resolve 'gray zone' transactions—cases where current deterministic systems lack sufficient context. The architecture leverages Event Sourcing and Domain-Driven Design (DDD) principles by building a centralized semantic layer that aggregates data from multiple bounded contexts (e.g., Transaction, Device, Account). Agents then consume this enriched context via an orchestration layer in a saga pattern to reach a final verdict.

Key takeaways

AI Agents Augment, Not Replace, Existing Systems
The primary value of AI agents is not replacing existing systems (like rule-based or ML engines) but rather adding an agentic layer to handle ambiguous cases—the 'gray zone' where current models lack necessary context. The goal is to enhance business workflows using the state and history already captured by event sourcing.
Context Aggregation via Semantic Layer 17:37
To provide sufficient context for agents, data from disparate bounded contexts (Transaction, Device, Account, Payment) must be aggregated into a semantic layer or materialized view. This is achieved by propagating domain events using Change Data Capture (CDC) mechanisms or message brokers.
Saga Orchestration and Agentic Decision Making
The system uses an orchestration layer to manage the process. This layer employs a fan-out pattern, sending events simultaneously to multiple specialized agents (e.g., Risk Analyzer Agent, Behavior Analyzer Agent). A final Verdict Agent then analyzes these responses to reach a consensus event.

Watch on YouTube Full article

Jul 29, 2026 · 16:29

SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe

Nubank achieved a massive acceleration in deploying AI agents—reporting up to 20x faster shipping—by shifting agent evaluation from relying solely on slow, expensive production data or manual curation to using grounded simulations. The core methodology involves running synthetic, multi-turn conversations against the agent within a specialized framework (Snowglobe), allowing teams to rapidly test models and variants in a continuous 'ship, observe, simulate, repeat' loop.

Key takeaways

Simulations Shortcircuit the Eval Bottleneck 10:44
Traditional agent evaluation data is complex (multi-turn, stateful trajectories) and time-consuming to gather via manual authoring or limited production traces. Simulations allow teams to generate thousands of multi-turn conversations quickly, reducing release cycles from weeks to hours.
The Simulation Loop: Ship, Observe, Simulate
A robust agent development cycle involves shipping the agent, observing real data, creating strong evaluations (using techniques like automated prompt optimization), and then using simulations to generate synthetic data. This simulated data is piped into the evaluation pipeline for continuous improvement.
Closing the Sim-to-Real Gap
To trust simulation results, it is crucial to establish metrics and processes that compare simulated performance against real production data (offline/online human review). The correlation between sim quality via evals and real data was found to be high.

Watch on YouTube Full article

Jul 29, 2026 · 14:35

We Scored Oracle's Database Skill Live: 95% Isn't Enough

The video demonstrates how Oracle's database team utilized AI 'skills'—encapsulated in a repository like `oracle/skills`—to guide AI agents in writing secure and best-practice compliant SQL. The process involved running an automated review (using Tessl) on a skill designed to improve database code generation, which initially scored 95%. Through iterative refinement and applying changes via a 'run, review, fix' cycle, the skill was successfully upgraded to a 100% success rate, showcasing a robust mechanism for encoding complex domain knowledge into AI workflows.

Key takeaways

Purpose of Database Skills 5:51
The primary function of the skill is to help agents write better SQL and database code by making them knowledgeable about the database's capabilities (e.g., property graphs, vector data). It helps discover what the database can do, rather than relying solely on general agent knowledge.
Skill Review Process 13:44
The skill was subjected to an automated review comparing it against best practices. The process involves running a 'run, review, fix' cycle, which applies changes and re-runs the validation in an agile style approach.
Achieving 100% Compliance 14:35
After making specific developer-suggested changes (e.g., adding a recommended sequence task), the skill's review score reached 100%, demonstrating continuous improvement and validation of domain knowledge.

Watch on YouTube Full article

Jul 29, 2026 · 18:26

Your Agent Didn't Fail. Your Harness Did. — Vinoth Govindarajan, OpenAI

The talk argues that most production failures in AI agents are not due to model hallucination or poor reasoning, but rather 'harness failures'—flaws in the underlying system architecture. The core principle is establishing a clear boundary: while a model can only propose an action, the surrounding harness must own the state transition, enforce ordered commits, and provide verifiable proof (the receipt) that the action was successfully executed and observed by the user.

Key takeaways

Model Proposes, Harness Commits, Receipt Proves 4:06
The model generates capability; the harness provides control. The system must own state transitions, enforce ordered mutations, and use a receipt to prove that an action was executed and visible to the user.
Own the State, Order the Mutation, Prove the Action 5:20
These three principles are critical for reliable agent systems. A fact must have a single owner (system of record) and a clear replay path; mutable state requires one ordered commit path.
The Run Receipt Audit
To audit an agent system, ask five questions: What woke it up (trigger)? What state did it inherit? Which authority did it use? What executed? And what evidence survived?

Watch on YouTube Full article

Jul 28, 2026 · 21:21

How Forward Deployed Engineering is done at Factory — Eno Reyes

Forward Deployed Engineering (FDE) is evolving from traditional professional services consulting into a strategic function where engineers act as 'the tip of the spear' for product development. The core goal is to build an autonomous 'Software Factory' within a customer's environment. This factory automates the entire signal-to-deploy pipeline—from capturing external signals (e.g., bug reports, Slack conversations) through planning, coding, and rigorous validation (SAST, linters, type checkers)—with minimal human intervention. Success hinges on achieving 'agent readiness,' which measures the density of deterministic validation loops in a codebase, enabling massive migrations (e.g., 30-50 million lines of code) autonomously.

Key takeaways

The Shift from Consulting to Product Strategy
Factory argues that FDE should not be professional services work on behalf of a customer, but rather the stream of information gathered from critical customers. This data is fed back into the product to rapidly improve it, making the deployed engineer an integral part of the product's evolution.
The Software Factory Pipeline 8:23
This factory concept describes a continuous process where external signals flow in, are prioritized into plans, converted into code changes (the source of truth), pass through automated validation stages (linters, type checkers, SAST tools), and finally deploy. The goal is an uninterrupted signal-to-deploy flow without human intervention.
Data Ownership and Autonomy 10:57
To ensure scalability and security, the deployed engineer must utilize a model independent agent harness (like Droid) that guarantees the customer owns all traces and data. This allows for highly secure deployments, including air-gapping in sensitive environments like finance or healthcare.
Agent Readiness as a Metric 20:05
Autonomy is measured by 'agent readiness,' which quantifies the number of deterministic validation loops (e.g., passing linters, type checkers) present in a codebase. A high volume of these loops allows advanced agents to operate on complex tasks for extended periods without human intervention.

Watch on YouTube Full article

Jul 28, 2026 · 20:23

AI tools for Forward Deployed Engineering — Vasuman Moza, Varick Agents

The core bottleneck in enterprise AI adoption is not model intelligence or execution capability, but rather the ability to deeply understand and re-engineer complex, undocumented human business processes. Forward Deployed Engineers (FDEs) are critical for this process, mapping existing workflows—including edge cases and handoffs—and building autonomous agents on top of legacy systems (e.g., SAP, NetSuite). The technical solution involves using a Veric OS platform that builds agents on existing systems without requiring costly migrations, leveraging dependency graphs, and employing custom post-trained models to extract accurate context from messy enterprise data.

Key takeaways

The AI Bottleneck is Context, Not Execution
While modern LLMs can solve the execution of work (intelligence constraint), the primary bottleneck remains understanding the unique business processes within a specific company. Every department (e.g., healthcare sales vs. SaaS sales) operates differently, requiring deep context extraction.
Forward Deployed Engineering (FDE) Role 7:24
FDEs are responsible for mapping how humans currently perform work and then re-engineering the process around AI. This ensures that AI solutions are adoptable and deliver measurable, department-wide ROI rather than failing as isolated 'point solutions.'
Non-Disruptive Deployment Strategy 17:02
To overcome enterprise resistance to migration (e.g., spending $5 million over five years on NetSuite), agents must be built *on top* of existing systems of record (Salesforce, SAP, Dynamics) rather than requiring a full system overhaul.
Agent Tooling for FDEs
The Veric platform provides specialized tools: an Engagement Agent (assistant to synthesize notes/docs), and a Workflow Agent that ensures the constructed workflow correctly shadows real-world process edge cases. A future autonomous assistant will handle minor changes without human intervention.

Watch on YouTube Full article

Jul 28, 2026 · 18:09

How Forward Deployed Engineering is done at Decagon — Sunny Rekhi

The talk details the function and evolution of Forward Deployed Engineering (FDE) at Decagon, a company specializing in AI customer service agents. FDE is described as being functionally identical to product engineering, requiring engineers to not only configure complex agent workflows for specific enterprises but also to proactively identify and build platform features that solve anticipated problems across multiple clients. The core philosophy emphasizes architectural restraint: ensuring all custom work is designed to be self-serve and compound into the overall platform, allowing the solution to scale from bespoke deployments to a generalized product.

Key takeaways

FDE Blurs with Product Engineering
The line between forward deployed service and internal product development is highly blurred. When an enterprise expresses a pain point, it should be treated as a potential product feature that needs prioritization for the entire platform (13:00).
Focus on Architectural Restraint 13:40
A critical skill in scaling FDE is exercising restraint—avoiding simple one-off patches and instead architecting solutions so they benefit future, unknown customers. This ensures the solution does not become a brittle 'black box' (8:20).
Custom Work Must Compound
The goal of every deployment is to ensure that custom integrations or workflows are systematically fed back into the platform, transforming bespoke solutions into self-service capabilities for all customers (14:18).

Watch on YouTube Full article

Jul 28, 2026 · 55:44

Introducing the SQL MCP Server - Jerry Nixon - NDC Toronto 2026

SQL MCP Server, an open-source component from Microsoft's data API builder, provides a secure and standardized abstraction layer for modern applications, especially those powered by AI agents. It allows external models to interact with enterprise databases (including SQL Server, PostgreSQL, Cosmos DB, and MySQL) without requiring direct T-SQL knowledge or exposing sensitive connection strings. The solution standardizes database access into three endpoints—REST, GraphQL, and MCP—simplifying the development process and significantly enhancing security by enforcing policies at the API layer.

Key takeaways

Unified Data Access Layer 2:00
The SQL MCP Server acts as a single point of entry for all data interactions, supporting multiple database types (SQL, NoSQL) and exposing three standardized APIs: REST, GraphQL, and the specialized Model Control Protocol (MCP). This abstraction layer allows developers to write code against the API surface rather than directly against the underlying schema.
AI Agent Integration via MCP 8:00
For AI agents, which cannot speak directly to a database, the MCP endpoint is critical. It collapses the potential hundreds of CRUD tools (one per table) into a standardized set of seven tools (Describe Entities, Create, Read, Update, Delete, etc.), preventing model overload and ensuring reliable agentic workflows.
Simplified Development & CI/CD 3:00
The solution aims to eliminate the need for boilerplate CRUD API code in a codebase. By using data API builder, developers can significantly reduce complexity and potential errors, streamlining the CI/CD pipeline while maintaining high functionality.
Advanced Security & Policy Enforcement 10:50
Security policies (like Row-Level Security) can be enforced at the API layer, even if structural changes are impossible in the underlying database. Furthermore, it supports advanced authentication methods like OpenID Connect and On Behalf Of (OBO) pass-through authentication.

Watch on YouTube Full article

Jul 28, 2026 · 58:39

Inside the Dark Factory: AI That Ships Code Solo

The video details Tessl's 'Dark Factory,' an autonomous system that handles a significant portion of their code shipping (65-70% of PRs). This factory uses AI agents and sophisticated verification layers to process tickets from inception (Linear) through to merge. The core architectural shift involves moving engineering focus from writing code to designing robust, automated workflows, emphasizing context management and layered verification to build trust in autonomous systems.

Key takeaways

High Automation Rate 2:24
Tessl routes 65-70% of its PRs through the Dark Factory. Historically, up to 95% of their codebase has never been reviewed by a human.
Shift in Engineering Focus 17:12
The role of the engineer shifts from implementing tickets to designing complex workflows, scoping tasks for agents, and encoding organizational 'taste' into verification layers (e.g., whiteboarding conversations).
Trust is Earned, Not Enabled 31:00
Building trust in the Dark Factory requires continuous effort, including running into failure modes and improving verification layers. Accountability remains with the person who filed the original ticket.
Verification Layers are Key 22:22
The most critical component is not the coding agent itself, but the layered verification system (Verifiers) that allow engineers to encode fuzzy principles (e.g., 'library should be the single source of truth') into deterministic checks.

Watch on YouTube Full article

Jul 28, 2026 · 10:36

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

The video compares Llama.cpp and vLLM, two leading engines for running Large Language Models (LLMs) locally. Llama.cpp is optimized for accessibility on consumer hardware (CPU/GPU), utilizing techniques like quantization and the GGUF format to run models efficiently on personal devices or edge environments. Conversely, vLLM focuses on maximizing efficiency at production scale, supporting diverse accelerators (NVIDIA, TPU, etc.) and implementing advanced optimizations such as continuous batching and paged attention for high-throughput workloads in cloud or Kubernetes deployments.

Key takeaways

Llama.cpp Use Case
Ideal for running LLMs on consumer hardware (laptops, Raspberry Pi) or edge devices due to its focus on accessibility and CPU/GPU optimization. Key features include quantization (reducing precision from FP16 to INT8/INT4) and packaging models into a single .gguf file.
vLLM Use Case 4:10
Designed for high-throughput, production-scale workloads in environments like VMs or Kubernetes. It supports diverse hardware accelerators (NVIDIA GPUs, TPUs, etc.) and advanced features like continuous batching and paged attention to manage KV cache efficiently.
Model Deployment Strategy 8:10
The choice depends on the environment: use Llama.cpp for personal/offline use cases, and vLLM when deploying in a high-performance, multi-user production setting.

Watch on YouTube Full article

Jul 28, 2026 · 33:39

AI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, Netflix

This talk outlines how Netflix leveraged AI agents to automate performance engineering, addressing the bottleneck of manually identifying and fixing inefficient code patterns in production services. The process involves feeding profiling data (call stacks, CPU usage) into an LLM agent, which identifies suboptimal algorithms (e.g., quadratic-time patterns). The agent then proposes a fix, validates it against a canary deployment using real production traffic, and can even scale the fix across multiple services by identifying common anti-patterns in a centralized catalog. This shifts performance optimization from a reactive, manual effort to a proactive, automated part of the SDLC.

Key takeaways

The Performance Bottleneck 4:00
Traditional performance engineering is highly manual and time-consuming (taking minutes just to identify hot paths), making it difficult to scale across large codebases. This bottleneck increases as coding agents write code faster, potentially introducing inefficiencies.
AI Agent Workflow for Optimization 12:17
An LLM agent can automate the entire loop: reading profiling data (call stack/CPU time), identifying anti-patterns (like O(N²) loops), locating the code path in the Git repo, proposing a fix, and validating it via canary deployment.
Scaling Fixes with Pattern Catalogs 21:20
Instead of fixing one instance, agents can search across multiple services using a centralized 'Pattern/Anti-pattern Catalog' (stored in a Git repo) to find and propose fixes for the same bad pattern repeatedly implemented elsewhere.
Shifting Left: Proactive Performance 26:40
The goal is to move performance improvement left in the SDLC. The catalog can be used proactively—at code review or even during code authoring—to suggest optimal patterns before suboptimal code reaches production.

Watch on YouTube Full article

Jul 27, 2026 · 53:34

Harness Engineering: Building an AI Software Factory

Harness engineering is a discipline focused on building automated loops of checks and maintenance agents that allow teams to delegate increasing amounts of codebase development to AI. The goal is not merely higher velocity, but achieving higher quality by shifting from manual code review (the primary bottleneck) to systematic process oversight. This involves tracking three key dimensions—autonomy, automation, and quality—and implementing layered validation systems: the Inner Loop (unit tests/linters), Outer Loop (agentic QA/UI testing), and Meta Loop (maintenance agents that analyze historical data for systemic improvements).

Key takeaways

The Three Dimensions of Agent Adoption 10:39
When adopting AI agents, track three metrics: Autonomy (how many human course corrections are needed); Automation (the level of oversight required, indicating trust); and Quality (ensuring the shipped product remains high quality). Progressing requires improving these dimensions sequentially.
The Three Loops of Harness Engineering 19:04
1. Inner Loop (Autonomy): Focuses on cheap, frequent checks like pedantic linting or unit tests to ensure agents get it right the first time. 2. Outer Loop (Automation): Involves slower, in-depth checks, such as agentic code review or running the product through a UI/CLI. 3. Meta Loop: Utilizes maintenance agents that analyze historical data (CI logs, PR comments) to propose systemic fixes and improvements to the entire process.
The Primary Barrier is Organizational 28:20
Harness engineering is fundamentally an organizational transformation, not just a technical one. Success requires changing workflows—for example, moving from monolithic PRs to smaller, low-risk chunks that can auto-merge, thereby shifting human behavior toward better practices.

Watch on YouTube Full article

Jul 25, 2026 · 33:15

Oleg Šelajev - You're absolutely right, it was your home directory! - AI Native DevCon June 2026

The session addresses the security risks posed by autonomous AI agents running locally, noting that these agents can pose significant threats by accessing sensitive data and executing arbitrary code. To mitigate this risk, Docker introduces sandboxing using MicroVMs—a hardware-level isolation primitive superior to traditional containers. This approach allows developers to run powerful AI agents in a restricted environment with controlled access to the filesystem, network, and secrets, enabling high productivity without sacrificing security.

Key takeaways

The Danger of Autonomous Agents 2:00
AI agents are highly autonomous and persistent; if given sufficient access (data, external communication, internet), they can execute malicious actions like stealing SSH keys or cloud credentials, leading to massive security incidents. The stakes are asymmetric: the attacker only needs to succeed once.
MicroVMs for Hard Isolation 6:00
Traditional containers share a kernel and are insufficient for high-stakes AI agent sandboxing due to potential container escaping exploits. Docker's solution uses MicroVMs, providing hardware-level isolation that significantly raises the security boundary.
The Role of Kits (YAML) 10:00
To improve developer experience in isolated environments, 'Kits' are introduced. These declarative YAML files define reusable configurations for a sandbox, specifying required commands, tools (e.g., programming language toolchains), file placements, and network rules.
Controlled Access Mechanisms 8:00
Sandboxes implement critical controls including networking proxies (to block specific domains/services) and secrets injection mechanisms, allowing agents to function without direct access to the host machine's private data.

Watch on YouTube Full article

Jul 25, 2026 · 20:24

From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

The talk outlines a shift in AI agent evaluation from simple production 'traces' to controlled, repeatable 'simulations.' To reliably develop and deploy agents, companies must build private benchmarks that mimic their specific production environment, tools, and policies. These simulations allow engineers to test the entire agent stack—including cost, latency, and policy adherence—and integrate this process into a formal CI pipeline for continuous improvement (Agent Ops).

Key takeaways

Private Benchmarks are Essential for Production Readiness 6:41
Public benchmarks (e.g., WebArena) are useful for general orientation but fail to capture domain-specific use cases, internal tooling, or company policies. A private benchmark is necessary because it allows comparison on critical metrics like cost per task and latency, not just success rate.
Simulation Enables Full Stack Evaluation 2:00
Offline simulation turns production traces into repeatable experiments. This allows testing the full agent stack (model, prompt, skills, tools) while keeping the environment and evaluators constant between runs, enabling apples-to-apples comparisons.
Benchmark Development Requires CI/CD Discipline
The benchmark itself must be treated as software. It requires a dedicated CI pipeline to ensure dependencies are pinned, fixtures are present, and the core Oracle solution passes before it can be used for agent testing.

Watch on YouTube Full article

Jul 24, 2026 · 20:36

From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize

The future of observability is shifting from human-driven dashboards to machine-readable telemetry that powers autonomous AI agents. Arize's Signal automates the debugging process by pulling deep production traces and logs (sometimes ten megabytes) directly into the repository as files. This allows coding harnesses, like Claude Code, to understand the exact code path taken during an error, enabling agents to propose fixes and creating a continuous loop where systems can autonomously improve themselves. The human role is evolving from responder to reviewer.

Key takeaways

Observability Shift (2.0) 2:16
Observability is moving beyond UI clicks and graphs; it's becoming a 'smoke'—telemetry data that agents can read to debug software, allowing for continuous automated fixing.
The Key Unlock: Traces on Filesystem 6:08
The critical breakthrough is pulling relevant production traces and logs down as files into the repo. Coding agents are highly effective with file formats, giving them the exact code path rather than guessing among millions of branches.
Autonomous Fixing Loop 6:54
The goal is to build systems that autonomously fix themselves. The process involves an agent investigating first, gathering deep evidence (traces/logs), and proposing a fix before human intervention.
Security and Deployment
To ensure compliance for large enterprises (e.g., Uber, Booking), agents must run within the customer's Virtual Private Cloud (VPC) using sandboxes, preventing production systems from connecting directly to external models.

Watch on YouTube Full article

Jul 24, 2026 · 6:06

The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

The complexity of modern AI agents—which now incorporate reasoning, tool calls, and long multi-step loops—has rendered traditional evaluation methods insufficient. The talk argues that the future of evaluating these systems lies in moving from static deterministic checks or fixed LLM rubrics to 'Agent as a Judge,' which performs adaptive dynamic analysis to uncover subtle failure modes.

Key takeaways

Evals are critical for AI maturity
Evals have become essential for serious AI teams, with the industry noting that they catch all failures and fuel continual learning loops. Arize reports running over 100 million evals monthly.
Agent complexity breaks traditional evals 3:26
As agents evolved from simple prompt answering (2023) to complex, multi-step loops with sub-agents and dynamic UI creation, the failure modes became fundamentally different, exceeding the scope of classical LLM as a Judge checks.
The future requires adaptive evaluation 5:45
While deterministic checks and LLM-as-a-Judge are valuable, the next step is 'Agent as a Judge,' which provides adaptive dynamic analysis to find failure modes that were previously undetectable.

Watch on YouTube Full article

Jul 24, 2026 · 54:33

Build Hour: Valuemaxxing with GPT-5.6

The session provides a deep dive into 'Value Maxxing' with GPT-5.6, shifting focus from merely maximizing token usage to optimizing AI agents for concrete business outcomes and cost efficiency. Key strategies include selecting the optimal model (Sol, Terra, Luna) based on workload needs, implementing advanced API features like programmatic tool calling and prompt caching, and structuring agent workflows to minimize redundant context processing.

Key takeaways

Shift from Token Maxxing to Value Maxxing 2:30
Progress should be measured by the value generated (e.g., time saved, quality improved), not just the number of tokens consumed. This requires defining clear outcomes and measurable 'good' for AI agents (evals).
Model Selection Strategy 4:09
The GPT-3.5 family includes Sol (flagship/complex tasks), Terra (balanced intelligence/cost/latency), and Luna (high-volume, cost/latency sensitive) to optimize for specific workloads.
Optimizing Agent Workflows with Codex 5:28
For day-to-day coding, starting with `GPT-5.6 Soul` on medium reasoning is often sufficient. Developers can also trade tokens for time using Fast mode or utilize Chronicle to build task memory.
Advanced API Techniques (Harness Optimization) 6:51
Implement programmatic tool calling (using a JavaScript sandbox) and prompt caching to significantly reduce input token costs and processing time. Persistent reasoning and compaction also boost performance and cache efficiency.

Watch on YouTube Full article

Jul 24, 2026 · 21:11

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

Alex Shaw introduces Harbor, a framework designed for evaluating and optimizing AI agents. The talk argues that agent development should be viewed through the lens of machine learning rather than traditional software engineering. This requires treating agent performance as a 'blackbox artifact' and managing it via empirical evaluation—a process formalized by 'rollouts.' Harbor provides the necessary infrastructure (sandboxes, standardized environments) to execute these complex evaluations in parallel.

Key takeaways

Agent Development vs. Software Engineering 5:15
Unlike traditional software engineering where behavior is predictable before execution, agentic coding and AI agents are best treated as blackbox artifacts whose performance requires empirical evaluation (e.g., 'Generated code is best treated as a blackbox artifact').
The Role of Rollouts in Agent Evaluation 10:30
Agent evaluation relies on 'rollouts' within sandboxed environments. This process involves passing the sandbox to the agent, collecting a trajectory, and then passing it to a verifier which produces rewards. Harbor standardizes this universal process.
Harbor as an Interoperable Standard 12:20
Harbor is presented as a common language and open-source framework for specifying agentic environments, allowing interoperability across different agents, models (e.g., GPT 5.5), and sandboxes to maximize data velocity.
Diverse Evaluation Use Cases 17:30
Evaluation can be highly specialized, including assessing how well agents build products (e.g., RampBench), how they use a product's headless mode, or automating internal processes.

Watch on YouTube Full article