Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
The presentation outlines the concept of building self-learning loops for AI agents, moving beyond traditional observability dashboards to programmatic interfaces that agents can consume and act upon. The core workflow involves detecting production failures (e.g., a broken price filter in a shopping assistant), investigating the root cause using real-time traces, proposing a fix, and finally, validating the fix against failure datasets in a continuous, automated loop. Arize provides tools like Signal and Arise Skills to facilitate this autonomous detection, investigation, and validation process.
Key takeaways
-
The Evolution of Observability
Software observability is moving from Phase 1 (human reading dashboards) to Phase 3 (fully autonomous loop) where the coding agent consumes telemetry events and self-corrects. Observability must evolve into programmatic interfaces that agents can operate on and self-reflect on.
-
Detect, Investigate, Fix, Verify Loop
17:37
The self-correcting loop involves: 1) **Detecting** failures (e.g., a price filter bug) before users report them. 2) **Investigating** the failure using real traces. 3) **Fixing** the issue (proposing a PR). 4) **Verifying** the fix against a dataset of actual failure cases.
-
Agent Experiments for Validation
28:10
Agent Experiments allow testing proposed fixes automatically via CI. By deploying the agent behind an endpoint, you can stream a dataset of actual failure traces into the fixed version and score it using a judge rubric, comparing performance before and after the fix.