The Video Signal

Topic

AI Agents

All digests tagged AI Agents

We Scored Oracle's Database Skill Live: 95% Isn't Enough thumbnail

· 14:35

We Scored Oracle's Database Skill Live: 95% Isn't Enough

The video demonstrates how Oracle's database team utilized AI 'skills'—encapsulated in a repository like `oracle/skills`—to guide AI agents in writing secure and best-practice compliant SQL. The process involved running an automated review (using Tessl) on a skill designed to improve database code generation, which initially scored 95%. Through iterative refinement and applying changes via a 'run, review, fix' cycle, the skill was successfully upgraded to a 100% success rate, showcasing a robust mechanism for encoding complex domain knowledge into AI workflows.

Key takeaways

  1. Purpose of Database Skills 5:51

    The primary function of the skill is to help agents write better SQL and database code by making them knowledgeable about the database's capabilities (e.g., property graphs, vector data). It helps discover what the database can do, rather than relying solely on general agent knowledge.

  2. Skill Review Process 13:44

    The skill was subjected to an automated review comparing it against best practices. The process involves running a 'run, review, fix' cycle, which applies changes and re-runs the validation in an agile style approach.

  3. Achieving 100% Compliance 14:35

    After making specific developer-suggested changes (e.g., adding a recommended sequence task), the skill's review score reached 100%, demonstrating continuous improvement and validation of domain knowledge.

Watch on YouTube Full article

Your Agent Didn't Fail. Your Harness Did. — Vinoth Govindarajan, OpenAI thumbnail

· 18:26

Your Agent Didn't Fail. Your Harness Did. — Vinoth Govindarajan, OpenAI

The talk argues that most production failures in AI agents are not due to model hallucination or poor reasoning, but rather 'harness failures'—flaws in the underlying system architecture. The core principle is establishing a clear boundary: while a model can only propose an action, the surrounding harness must own the state transition, enforce ordered commits, and provide verifiable proof (the receipt) that the action was successfully executed and observed by the user.

Key takeaways

  1. Model Proposes, Harness Commits, Receipt Proves 4:06

    The model generates capability; the harness provides control. The system must own state transitions, enforce ordered mutations, and use a receipt to prove that an action was executed and visible to the user.

  2. Own the State, Order the Mutation, Prove the Action 5:20

    These three principles are critical for reliable agent systems. A fact must have a single owner (system of record) and a clear replay path; mutable state requires one ordered commit path.

  3. The Run Receipt Audit

    To audit an agent system, ask five questions: What woke it up (trigger)? What state did it inherit? Which authority did it use? What executed? And what evidence survived?

Watch on YouTube Full article

How Forward Deployed Engineering is done at Factory — Eno Reyes thumbnail

· 21:21

How Forward Deployed Engineering is done at Factory — Eno Reyes

Forward Deployed Engineering (FDE) is evolving from traditional professional services consulting into a strategic function where engineers act as 'the tip of the spear' for product development. The core goal is to build an autonomous 'Software Factory' within a customer's environment. This factory automates the entire signal-to-deploy pipeline—from capturing external signals (e.g., bug reports, Slack conversations) through planning, coding, and rigorous validation (SAST, linters, type checkers)—with minimal human intervention. Success hinges on achieving 'agent readiness,' which measures the density of deterministic validation loops in a codebase, enabling massive migrations (e.g., 30-50 million lines of code) autonomously.

Key takeaways

  1. The Shift from Consulting to Product Strategy

    Factory argues that FDE should not be professional services work on behalf of a customer, but rather the stream of information gathered from critical customers. This data is fed back into the product to rapidly improve it, making the deployed engineer an integral part of the product's evolution.

  2. The Software Factory Pipeline 8:23

    This factory concept describes a continuous process where external signals flow in, are prioritized into plans, converted into code changes (the source of truth), pass through automated validation stages (linters, type checkers, SAST tools), and finally deploy. The goal is an uninterrupted signal-to-deploy flow without human intervention.

  3. Data Ownership and Autonomy 10:57

    To ensure scalability and security, the deployed engineer must utilize a model independent agent harness (like Droid) that guarantees the customer owns all traces and data. This allows for highly secure deployments, including air-gapping in sensitive environments like finance or healthcare.

  4. Agent Readiness as a Metric 20:05

    Autonomy is measured by 'agent readiness,' which quantifies the number of deterministic validation loops (e.g., passing linters, type checkers) present in a codebase. A high volume of these loops allows advanced agents to operate on complex tasks for extended periods without human intervention.

Watch on YouTube Full article

AI tools for Forward Deployed Engineering — Vasuman Moza, Varick Agents thumbnail

· 20:23

AI tools for Forward Deployed Engineering — Vasuman Moza, Varick Agents

The core bottleneck in enterprise AI adoption is not model intelligence or execution capability, but rather the ability to deeply understand and re-engineer complex, undocumented human business processes. Forward Deployed Engineers (FDEs) are critical for this process, mapping existing workflows—including edge cases and handoffs—and building autonomous agents on top of legacy systems (e.g., SAP, NetSuite). The technical solution involves using a Veric OS platform that builds agents on existing systems without requiring costly migrations, leveraging dependency graphs, and employing custom post-trained models to extract accurate context from messy enterprise data.

Key takeaways

  1. The AI Bottleneck is Context, Not Execution

    While modern LLMs can solve the execution of work (intelligence constraint), the primary bottleneck remains understanding the unique business processes within a specific company. Every department (e.g., healthcare sales vs. SaaS sales) operates differently, requiring deep context extraction.

  2. Forward Deployed Engineering (FDE) Role 7:24

    FDEs are responsible for mapping how humans currently perform work and then re-engineering the process around AI. This ensures that AI solutions are adoptable and deliver measurable, department-wide ROI rather than failing as isolated 'point solutions.'

  3. Non-Disruptive Deployment Strategy 17:02

    To overcome enterprise resistance to migration (e.g., spending $5 million over five years on NetSuite), agents must be built *on top* of existing systems of record (Salesforce, SAP, Dynamics) rather than requiring a full system overhaul.

  4. Agent Tooling for FDEs

    The Veric platform provides specialized tools: an Engagement Agent (assistant to synthesize notes/docs), and a Workflow Agent that ensures the constructed workflow correctly shadows real-world process edge cases. A future autonomous assistant will handle minor changes without human intervention.

Watch on YouTube Full article

How Forward Deployed Engineering is done at Decagon — Sunny Rekhi thumbnail

· 18:09

How Forward Deployed Engineering is done at Decagon — Sunny Rekhi

The talk details the function and evolution of Forward Deployed Engineering (FDE) at Decagon, a company specializing in AI customer service agents. FDE is described as being functionally identical to product engineering, requiring engineers to not only configure complex agent workflows for specific enterprises but also to proactively identify and build platform features that solve anticipated problems across multiple clients. The core philosophy emphasizes architectural restraint: ensuring all custom work is designed to be self-serve and compound into the overall platform, allowing the solution to scale from bespoke deployments to a generalized product.

Key takeaways

  1. FDE Blurs with Product Engineering

    The line between forward deployed service and internal product development is highly blurred. When an enterprise expresses a pain point, it should be treated as a potential product feature that needs prioritization for the entire platform (13:00).

  2. Focus on Architectural Restraint 13:40

    A critical skill in scaling FDE is exercising restraint—avoiding simple one-off patches and instead architecting solutions so they benefit future, unknown customers. This ensures the solution does not become a brittle 'black box' (8:20).

  3. Custom Work Must Compound

    The goal of every deployment is to ensure that custom integrations or workflows are systematically fed back into the platform, transforming bespoke solutions into self-service capabilities for all customers (14:18).

Watch on YouTube Full article

Introducing the SQL MCP Server - Jerry Nixon - NDC Toronto 2026 thumbnail

· 55:44

Introducing the SQL MCP Server - Jerry Nixon - NDC Toronto 2026

SQL MCP Server, an open-source component from Microsoft's data API builder, provides a secure and standardized abstraction layer for modern applications, especially those powered by AI agents. It allows external models to interact with enterprise databases (including SQL Server, PostgreSQL, Cosmos DB, and MySQL) without requiring direct T-SQL knowledge or exposing sensitive connection strings. The solution standardizes database access into three endpoints—REST, GraphQL, and MCP—simplifying the development process and significantly enhancing security by enforcing policies at the API layer.

Key takeaways

  1. Unified Data Access Layer 2:00

    The SQL MCP Server acts as a single point of entry for all data interactions, supporting multiple database types (SQL, NoSQL) and exposing three standardized APIs: REST, GraphQL, and the specialized Model Control Protocol (MCP). This abstraction layer allows developers to write code against the API surface rather than directly against the underlying schema.

  2. AI Agent Integration via MCP 8:00

    For AI agents, which cannot speak directly to a database, the MCP endpoint is critical. It collapses the potential hundreds of CRUD tools (one per table) into a standardized set of seven tools (Describe Entities, Create, Read, Update, Delete, etc.), preventing model overload and ensuring reliable agentic workflows.

  3. Simplified Development & CI/CD 3:00

    The solution aims to eliminate the need for boilerplate CRUD API code in a codebase. By using data API builder, developers can significantly reduce complexity and potential errors, streamlining the CI/CD pipeline while maintaining high functionality.

  4. Advanced Security & Policy Enforcement 10:50

    Security policies (like Row-Level Security) can be enforced at the API layer, even if structural changes are impossible in the underlying database. Furthermore, it supports advanced authentication methods like OpenID Connect and On Behalf Of (OBO) pass-through authentication.

Watch on YouTube Full article

Inside the Dark Factory: AI That Ships Code Solo thumbnail

· 58:39

Inside the Dark Factory: AI That Ships Code Solo

The video details Tessl's 'Dark Factory,' an autonomous system that handles a significant portion of their code shipping (65-70% of PRs). This factory uses AI agents and sophisticated verification layers to process tickets from inception (Linear) through to merge. The core architectural shift involves moving engineering focus from writing code to designing robust, automated workflows, emphasizing context management and layered verification to build trust in autonomous systems.

Key takeaways

  1. High Automation Rate 2:24

    Tessl routes 65-70% of its PRs through the Dark Factory. Historically, up to 95% of their codebase has never been reviewed by a human.

  2. Shift in Engineering Focus 17:12

    The role of the engineer shifts from implementing tickets to designing complex workflows, scoping tasks for agents, and encoding organizational 'taste' into verification layers (e.g., whiteboarding conversations).

  3. Trust is Earned, Not Enabled 31:00

    Building trust in the Dark Factory requires continuous effort, including running into failure modes and improving verification layers. Accountability remains with the person who filed the original ticket.

  4. Verification Layers are Key 22:22

    The most critical component is not the coding agent itself, but the layered verification system (Verifiers) that allow engineers to encode fuzzy principles (e.g., 'library should be the single source of truth') into deterministic checks.

Watch on YouTube Full article

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales? thumbnail

· 10:36

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

The video compares Llama.cpp and vLLM, two leading engines for running Large Language Models (LLMs) locally. Llama.cpp is optimized for accessibility on consumer hardware (CPU/GPU), utilizing techniques like quantization and the GGUF format to run models efficiently on personal devices or edge environments. Conversely, vLLM focuses on maximizing efficiency at production scale, supporting diverse accelerators (NVIDIA, TPU, etc.) and implementing advanced optimizations such as continuous batching and paged attention for high-throughput workloads in cloud or Kubernetes deployments.

Key takeaways

  1. Llama.cpp Use Case

    Ideal for running LLMs on consumer hardware (laptops, Raspberry Pi) or edge devices due to its focus on accessibility and CPU/GPU optimization. Key features include quantization (reducing precision from FP16 to INT8/INT4) and packaging models into a single .gguf file.

  2. vLLM Use Case 4:10

    Designed for high-throughput, production-scale workloads in environments like VMs or Kubernetes. It supports diverse hardware accelerators (NVIDIA GPUs, TPUs, etc.) and advanced features like continuous batching and paged attention to manage KV cache efficiently.

  3. Model Deployment Strategy 8:10

    The choice depends on the environment: use Llama.cpp for personal/offline use cases, and vLLM when deploying in a high-performance, multi-user production setting.

Watch on YouTube Full article

AI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, Netflix thumbnail

· 33:39

AI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, Netflix

This talk outlines how Netflix leveraged AI agents to automate performance engineering, addressing the bottleneck of manually identifying and fixing inefficient code patterns in production services. The process involves feeding profiling data (call stacks, CPU usage) into an LLM agent, which identifies suboptimal algorithms (e.g., quadratic-time patterns). The agent then proposes a fix, validates it against a canary deployment using real production traffic, and can even scale the fix across multiple services by identifying common anti-patterns in a centralized catalog. This shifts performance optimization from a reactive, manual effort to a proactive, automated part of the SDLC.

Key takeaways

  1. The Performance Bottleneck 4:00

    Traditional performance engineering is highly manual and time-consuming (taking minutes just to identify hot paths), making it difficult to scale across large codebases. This bottleneck increases as coding agents write code faster, potentially introducing inefficiencies.

  2. AI Agent Workflow for Optimization 12:17

    An LLM agent can automate the entire loop: reading profiling data (call stack/CPU time), identifying anti-patterns (like O(N²) loops), locating the code path in the Git repo, proposing a fix, and validating it via canary deployment.

  3. Scaling Fixes with Pattern Catalogs 21:20

    Instead of fixing one instance, agents can search across multiple services using a centralized 'Pattern/Anti-pattern Catalog' (stored in a Git repo) to find and propose fixes for the same bad pattern repeatedly implemented elsewhere.

  4. Shifting Left: Proactive Performance 26:40

    The goal is to move performance improvement left in the SDLC. The catalog can be used proactively—at code review or even during code authoring—to suggest optimal patterns before suboptimal code reaches production.

Watch on YouTube Full article

Harness Engineering: Building an AI Software Factory thumbnail

· 53:34

Harness Engineering: Building an AI Software Factory

Harness engineering is a discipline focused on building automated loops of checks and maintenance agents that allow teams to delegate increasing amounts of codebase development to AI. The goal is not merely higher velocity, but achieving higher quality by shifting from manual code review (the primary bottleneck) to systematic process oversight. This involves tracking three key dimensions—autonomy, automation, and quality—and implementing layered validation systems: the Inner Loop (unit tests/linters), Outer Loop (agentic QA/UI testing), and Meta Loop (maintenance agents that analyze historical data for systemic improvements).

Key takeaways

  1. The Three Dimensions of Agent Adoption 10:39

    When adopting AI agents, track three metrics: Autonomy (how many human course corrections are needed); Automation (the level of oversight required, indicating trust); and Quality (ensuring the shipped product remains high quality). Progressing requires improving these dimensions sequentially.

  2. The Three Loops of Harness Engineering 19:04

    1. Inner Loop (Autonomy): Focuses on cheap, frequent checks like pedantic linting or unit tests to ensure agents get it right the first time. 2. Outer Loop (Automation): Involves slower, in-depth checks, such as agentic code review or running the product through a UI/CLI. 3. Meta Loop: Utilizes maintenance agents that analyze historical data (CI logs, PR comments) to propose systemic fixes and improvements to the entire process.

  3. The Primary Barrier is Organizational 28:20

    Harness engineering is fundamentally an organizational transformation, not just a technical one. Success requires changing workflows—for example, moving from monolithic PRs to smaller, low-risk chunks that can auto-merge, thereby shifting human behavior toward better practices.

Watch on YouTube Full article

Simon Obstbaum & Rob Willoughby - Why evals are hard and how we're solving it - AI Native DevCon Jun thumbnail

· 36:39

Simon Obstbaum & Rob Willoughby - Why evals are hard and how we're solving it - AI Native DevCon Jun

This session introduces advanced methods for evaluating AI agents, arguing that relying solely on 'output evals' (what came out) is insufficient. The focus must shift to 'trajectory evals,' which measure whether the agent followed the correct steps and utilized the right tools. By instrumenting agent behavior—specifically through structured skills and context—teams can significantly improve metrics like PR throughput, decrease cognitive complexity, and ensure adherence to unique organizational conventions (e.g., internal API choices or security policies).

Key takeaways

  1. Shift from Output Evals to Trajectory Evals 29:56

    Evaluating agents requires measuring not just the final output, but also whether the agent activated the correct skills and followed the intended workflow (trajectory) [0:35:46]. Separating activation, trajectory, and outcome is essential for optimizing performance.

  2. Structured Context Improves Code Quality 5:46

    The analysis shows that moving from unstructured (L1) to structured context (L3) significantly improves code quality metrics. Specifically, L2 and L3 teams show increased PR throughput, decreased revert rates, and lower cognitive complexity compared to L1 [0:58:46].

  3. Instruction Following is the Key Differentiator 6:32

    While task completion may remain high regardless of structure, 'instruction following' (grounded in skills) measures adherence to unique organizational rules. This metric shows the biggest lift and represents the value of encoding proprietary business IP into the agent's context [1:03:52].

  4. The System, Not Just the Model, Matters 10:52

    Performance is highly dependent on the entire system stack. Testing must account for the specific model harness (e.g., Opus 4 8 in Claude Code vs. OpenHands), as changing the harness can move scores by up to 100% [1:09:52].

Watch on YouTube Full article

Oleg Šelajev - You're absolutely right, it was your home directory! - AI Native DevCon June 2026 thumbnail

· 33:15

Oleg Šelajev - You're absolutely right, it was your home directory! - AI Native DevCon June 2026

The session addresses the security risks posed by autonomous AI agents running locally, noting that these agents can pose significant threats by accessing sensitive data and executing arbitrary code. To mitigate this risk, Docker introduces sandboxing using MicroVMs—a hardware-level isolation primitive superior to traditional containers. This approach allows developers to run powerful AI agents in a restricted environment with controlled access to the filesystem, network, and secrets, enabling high productivity without sacrificing security.

Key takeaways

  1. The Danger of Autonomous Agents 2:00

    AI agents are highly autonomous and persistent; if given sufficient access (data, external communication, internet), they can execute malicious actions like stealing SSH keys or cloud credentials, leading to massive security incidents. The stakes are asymmetric: the attacker only needs to succeed once.

  2. MicroVMs for Hard Isolation 6:00

    Traditional containers share a kernel and are insufficient for high-stakes AI agent sandboxing due to potential container escaping exploits. Docker's solution uses MicroVMs, providing hardware-level isolation that significantly raises the security boundary.

  3. The Role of Kits (YAML) 10:00

    To improve developer experience in isolated environments, 'Kits' are introduced. These declarative YAML files define reusable configurations for a sandbox, specifying required commands, tools (e.g., programming language toolchains), file placements, and network rules.

  4. Controlled Access Mechanisms 8:00

    Sandboxes implement critical controls including networking proxies (to block specific domains/services) and secrets injection mechanisms, allowing agents to function without direct access to the host machine's private data.

Watch on YouTube Full article

From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI thumbnail

· 20:24

From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

The talk outlines a shift in AI agent evaluation from simple production 'traces' to controlled, repeatable 'simulations.' To reliably develop and deploy agents, companies must build private benchmarks that mimic their specific production environment, tools, and policies. These simulations allow engineers to test the entire agent stack—including cost, latency, and policy adherence—and integrate this process into a formal CI pipeline for continuous improvement (Agent Ops).

Key takeaways

  1. Private Benchmarks are Essential for Production Readiness 6:41

    Public benchmarks (e.g., WebArena) are useful for general orientation but fail to capture domain-specific use cases, internal tooling, or company policies. A private benchmark is necessary because it allows comparison on critical metrics like cost per task and latency, not just success rate.

  2. Simulation Enables Full Stack Evaluation 2:00

    Offline simulation turns production traces into repeatable experiments. This allows testing the full agent stack (model, prompt, skills, tools) while keeping the environment and evaluators constant between runs, enabling apples-to-apples comparisons.

  3. Benchmark Development Requires CI/CD Discipline

    The benchmark itself must be treated as software. It requires a dedicated CI pipeline to ensure dependencies are pinned, fixtures are present, and the core Oracle solution passes before it can be used for agent testing.

Watch on YouTube Full article

From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize thumbnail

· 20:36

From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize

The future of observability is shifting from human-driven dashboards to machine-readable telemetry that powers autonomous AI agents. Arize's Signal automates the debugging process by pulling deep production traces and logs (sometimes ten megabytes) directly into the repository as files. This allows coding harnesses, like Claude Code, to understand the exact code path taken during an error, enabling agents to propose fixes and creating a continuous loop where systems can autonomously improve themselves. The human role is evolving from responder to reviewer.

Key takeaways

  1. Observability Shift (2.0) 2:16

    Observability is moving beyond UI clicks and graphs; it's becoming a 'smoke'—telemetry data that agents can read to debug software, allowing for continuous automated fixing.

  2. The Key Unlock: Traces on Filesystem 6:08

    The critical breakthrough is pulling relevant production traces and logs down as files into the repo. Coding agents are highly effective with file formats, giving them the exact code path rather than guessing among millions of branches.

  3. Autonomous Fixing Loop 6:54

    The goal is to build systems that autonomously fix themselves. The process involves an agent investigating first, gathering deep evidence (traces/logs), and proposing a fix before human intervention.

  4. Security and Deployment

    To ensure compliance for large enterprises (e.g., Uber, Booking), agents must run within the customer's Virtual Private Cloud (VPC) using sandboxes, preventing production systems from connecting directly to external models.

Watch on YouTube Full article

The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI thumbnail

· 6:06

The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

The complexity of modern AI agents—which now incorporate reasoning, tool calls, and long multi-step loops—has rendered traditional evaluation methods insufficient. The talk argues that the future of evaluating these systems lies in moving from static deterministic checks or fixed LLM rubrics to 'Agent as a Judge,' which performs adaptive dynamic analysis to uncover subtle failure modes.

Key takeaways

  1. Evals are critical for AI maturity

    Evals have become essential for serious AI teams, with the industry noting that they catch all failures and fuel continual learning loops. Arize reports running over 100 million evals monthly.

  2. Agent complexity breaks traditional evals 3:26

    As agents evolved from simple prompt answering (2023) to complex, multi-step loops with sub-agents and dynamic UI creation, the failure modes became fundamentally different, exceeding the scope of classical LLM as a Judge checks.

  3. The future requires adaptive evaluation 5:45

    While deterministic checks and LLM-as-a-Judge are valuable, the next step is 'Agent as a Judge,' which provides adaptive dynamic analysis to find failure modes that were previously undetectable.

Watch on YouTube Full article

Build Hour: Valuemaxxing with GPT-5.6 thumbnail

· 54:33

Build Hour: Valuemaxxing with GPT-5.6

The session provides a deep dive into 'Value Maxxing' with GPT-5.6, shifting focus from merely maximizing token usage to optimizing AI agents for concrete business outcomes and cost efficiency. Key strategies include selecting the optimal model (Sol, Terra, Luna) based on workload needs, implementing advanced API features like programmatic tool calling and prompt caching, and structuring agent workflows to minimize redundant context processing.

Key takeaways

  1. Shift from Token Maxxing to Value Maxxing 2:30

    Progress should be measured by the value generated (e.g., time saved, quality improved), not just the number of tokens consumed. This requires defining clear outcomes and measurable 'good' for AI agents (evals).

  2. Model Selection Strategy 4:09

    The GPT-3.5 family includes Sol (flagship/complex tasks), Terra (balanced intelligence/cost/latency), and Luna (high-volume, cost/latency sensitive) to optimize for specific workloads.

  3. Optimizing Agent Workflows with Codex 5:28

    For day-to-day coding, starting with `GPT-5.6 Soul` on medium reasoning is often sufficient. Developers can also trade tokens for time using Fast mode or utilize Chronicle to build task memory.

  4. Advanced API Techniques (Harness Optimization) 6:51

    Implement programmatic tool calling (using a JavaScript sandbox) and prompt caching to significantly reduce input token costs and processing time. Persistent reasoning and compaction also boost performance and cache efficiency.

Watch on YouTube Full article

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute thumbnail

· 21:11

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

Alex Shaw introduces Harbor, a framework designed for evaluating and optimizing AI agents. The talk argues that agent development should be viewed through the lens of machine learning rather than traditional software engineering. This requires treating agent performance as a 'blackbox artifact' and managing it via empirical evaluation—a process formalized by 'rollouts.' Harbor provides the necessary infrastructure (sandboxes, standardized environments) to execute these complex evaluations in parallel.

Key takeaways

  1. Agent Development vs. Software Engineering 5:15

    Unlike traditional software engineering where behavior is predictable before execution, agentic coding and AI agents are best treated as blackbox artifacts whose performance requires empirical evaluation (e.g., 'Generated code is best treated as a blackbox artifact').

  2. The Role of Rollouts in Agent Evaluation 10:30

    Agent evaluation relies on 'rollouts' within sandboxed environments. This process involves passing the sandbox to the agent, collecting a trajectory, and then passing it to a verifier which produces rewards. Harbor standardizes this universal process.

  3. Harbor as an Interoperable Standard 12:20

    Harbor is presented as a common language and open-source framework for specifying agentic environments, allowing interoperability across different agents, models (e.g., GPT 5.5), and sandboxes to maximize data velocity.

  4. Diverse Evaluation Use Cases 17:30

    Evaluation can be highly specialized, including assessing how well agents build products (e.g., RampBench), how they use a product's headless mode, or automating internal processes.

Watch on YouTube Full article

Full Workshop: Setting Yourself Up for Success —Jason Liu, OpenAI Codex thumbnail

· 1:15:02

Full Workshop: Setting Yourself Up for Success —Jason Liu, OpenAI Codex

Jason Liu provides a comprehensive workshop on leveraging Codex and the Agents SDK for advanced knowledge work automation. The focus shifts from simple text generation to using AI agents (threads) as collaborative teammates capable of managing complex projects, interacting with external applications via 'Computer Use,' and maintaining long-running processes through automated 'heartbeats.' Key strategies include building custom skills/plugins, utilizing a personal memory vault, and treating the computer itself as an integrated component of the workflow.

Key takeaways

  1. Systematic Context Management via Appshots 20:05

    The 'Appshot' feature is highlighted as critical for providing deep context to AI models. Unlike simple screenshots, Appshots capture the entire accessibility tree of an application (e.g., Slack), allowing agents to perform complex actions with single function calls and greatly improving accuracy.

  2. Advanced Project Management with Threads 5:50

    The concept of 'compaction' allows threads (pinned conversations) to maintain long-term context, acting like dedicated project managers. These threads can delegate tasks to sub-agents and communicate with each other, forming a collaborative team structure.

  3. Automating Long-Running Workflows (Heartbeats) 11:30

    Instead of manual checks, 'heartbeat' automations can wake up threads over time to perform continuous maintenance tasks (e.g., checking PR status, monitoring support issues), allowing for sustained project oversight.

  4. Extending AI Capabilities with Skills and Plugins 23:25

    Skills are simple constructs (files/scripts) that solve specific problems (e.g., 'Review my code like Charlie'). Plugins connect the system to external services (Slack, Gmail), allowing agents to perform actions across different platforms.

Watch on YouTube Full article

Why AI Agents Need a Data Harness, Not Just a Lakehouse thumbnail

· 37:07

Why AI Agents Need a Data Harness, Not Just a Lakehouse

The talk outlines the necessity of a 'data harness'—a modern data foundation optimized for conversational-pace querying by AI agents. Traditional data pipelines built for minutes-to-hours turnaround fail under this rapid load, leading to errors and wasted tokens. The solution requires combining an open lakehouse architecture (using standards like Apache Iceberg and Apache Arrow) with a robust semantic layer, advanced caching (like Columnar Cloud Cache/C3), and federation capabilities to ensure data is accessible, understandable, and performant for AI use cases.

Key takeaways

  1. AI Agents Require Subsecond Performance 6:32

    Conversational analytics moves at a conversational pace. Legacy systems designed for minutes-to-hours turnaround cannot support this speed; subsecond response time is a critical requirement for agentic workflows.

  2. Three Pillars of AI-Ready Data Foundation 9:32

    Any modern data foundation must be accessible (seeing the whole estate), understandable (having clear, standardized definitions via a semantic layer), and performant (querying data where it lives without moving or cleaning it).

  3. Open Standards Prevent Vendor Lock-in 24:12

    The use of open standards like Apache Iceberg, Apache Arrow, and Apache Polaris ensures that the data platform remains vendor-neutral. This allows users to build performance without sacrificing portability.

Watch on YouTube Full article

Perception Agents — Antje Barth, Amazon AGI Lab thumbnail

· 21:45

Perception Agents — Antje Barth, Amazon AGI Lab

Current AI agents excel at discrete tasks like clicking or calling APIs but fail in complex, end-to-end knowledge work because they lack reliable perception and verification capabilities. The talk introduces 'Perception Agents,' which close the architectural gap by enabling agents to perceive rendered UIs (not just underlying code), maintain shared context, and verify their own output against design specs or user flows, mimicking human collaboration.

Key takeaways

  1. The Gap in Agent Capability

    Current agents struggle with end-to-end workflows because the 'real work' lives within the seams of multiple applications. While they can perform individual steps, they cannot manage the full process reliability required for critical tasks (e.g., deleting a database).

  2. The Need for Reliability and Verification 3:50

    Unlike code, which is verifiable through unit tests, most knowledge work is 'messy' and lacks easy verification methods. This lack of verifiability is the primary hurdle to building trust in agents.

  3. Perception Agents: Closing the Loop 7:40

    A perception agent must perceive the screen (rendered layout, state) like a human, not just scrape code. They must complete the loop by observing results to confirm if actions succeeded, rather than simply firing off commands.

  4. Shared Context and Multimodal Perception 10:40

    Perception is more than just visual input; it includes understanding context from sources like audio transcripts. The goal is to build agents that react in real-time, similar to human collaboration, without the back-and-forth of prompt/response cycles.

Watch on YouTube Full article