Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Summary
The presentation outlines the concept of building self-learning loops for AI agents, moving beyond traditional observability dashboards to programmatic interfaces that agents can consume and act upon. The core workflow involves detecting production failures (e.g., a broken price filter in a shopping assistant), investigating the root cause using real-time traces, proposing a fix, and finally, validating the fix against failure datasets in a continuous, automated loop. Arize provides tools like Signal and Arise Skills to facilitate this autonomous detection, investigation, and validation process.
Key takeaways
-
The Evolution of Observability
Software observability is moving from Phase 1 (human reading dashboards) to Phase 3 (fully autonomous loop) where the coding agent consumes telemetry events and self-corrects. Observability must evolve into programmatic interfaces that agents can operate on and self-reflect on.
-
Detect, Investigate, Fix, Verify Loop
17:37
The self-correcting loop involves: 1) **Detecting** failures (e.g., a price filter bug) before users report them. 2) **Investigating** the failure using real traces. 3) **Fixing** the issue (proposing a PR). 4) **Verifying** the fix against a dataset of actual failure cases.
-
Agent Experiments for Validation
28:10
Agent Experiments allow testing proposed fixes automatically via CI. By deploying the agent behind an endpoint, you can stream a dataset of actual failure traces into the fixed version and score it using a judge rubric, comparing performance before and after the fix.
Technical details
-
Arise Signal
919s
Signal is an out-of-the-box agent that continuously runs on production traces. It clusters failures by root cause, connects to external tools (like GitHub and DataDog), and self-reflects on the agent's code to write investigation reports and propose Pull Requests (PRs).
-
Arise Skills
1057s
Arise Skills allow external coding agents (like Cursor, Claude, or CodeX) to query traces directly from the Arise platform. This provides the coding agent with both the actual code and the real-time trace data simultaneously, enabling more impactful fixes.
-
Harness as a Judge
1890s
This advanced evaluation method moves beyond fixed rubrics (like LM as a judge). It performs adaptive analysis by reading the actual trace tree directly, allowing it to score unknown unknowns (e.g., checking if a returned product price satisfies a requested range) and verifying complex logic like access checks.
-
Setup and Workflow
1330s
The basic setup requires cloning the repository, running `npm install`, and configuring the `M` file with both the Arise API key and the OpenAI API key to enable tracing and agent functionality.
Mentioned resources
- Arise AI
- OpenAI Agents SDK
- Arize Skills
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.