# Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI

## Executive summary

The presentation outlines the concept of building self-learning loops for AI agents, moving beyond traditional observability dashboards to programmatic interfaces that agents can consume and act upon. The core workflow involves detecting production failures (e.g., a broken price filter in a shopping assistant), investigating the root cause using real-time traces, proposing a fix, and finally, validating the fix against failure datasets in a continuous, automated loop. Arize provides tools like Signal and Arise Skills to facilitate this autonomous detection, investigation, and validation process.

## Key takeaways

- The Evolution of Observability: Software observability is moving from Phase 1 (human reading dashboards) to Phase 3 (fully autonomous loop) where the coding agent consumes telemetry events and self-corrects. Observability must evolve into programmatic interfaces that agents can operate on and self-reflect on.
- Detect, Investigate, Fix, Verify Loop: The self-correcting loop involves: 1) **Detecting** failures (e.g., a price filter bug) before users report them. 2) **Investigating** the failure using real traces. 3) **Fixing** the issue (proposing a PR). 4) **Verifying** the fix against a dataset of actual failure cases.
- Agent Experiments for Validation: Agent Experiments allow testing proposed fixes automatically via CI. By deploying the agent behind an endpoint, you can stream a dataset of actual failure traces into the fixed version and score it using a judge rubric, comparing performance before and after the fix.

## Technical details

- Arise Signal: Signal is an out-of-the-box agent that continuously runs on production traces. It clusters failures by root cause, connects to external tools (like GitHub and DataDog), and self-reflects on the agent's code to write investigation reports and propose Pull Requests (PRs).
- Arise Skills: Arise Skills allow external coding agents (like Cursor, Claude, or CodeX) to query traces directly from the Arise platform. This provides the coding agent with both the actual code and the real-time trace data simultaneously, enabling more impactful fixes.
- Harness as a Judge: This advanced evaluation method moves beyond fixed rubrics (like LM as a judge). It performs adaptive analysis by reading the actual trace tree directly, allowing it to score unknown unknowns (e.g., checking if a returned product price satisfies a requested range) and verifying complex logic like access checks.
- Setup and Workflow: The basic setup requires cloning the repository, running `npm install`, and configuring the `M` file with both the Arise API key and the OpenAI API key to enable tracing and agent functionality.

## Practical implications

- Implement continuous monitoring agents (like Signal) to automatically detect and investigate production failures, reducing reliance on manual dashboard inspection.
- Integrate observability data (traces) directly into your agent's context to enable self-correction and propose code fixes.
- Use Agent Experiments and Harness as a Judge for automated, adaptive validation of fixes against real-world failure datasets before deployment.

## Topics

AI Agents, Observability, Self-Healing Software, Continuous Integration (CI), LLM Evaluation, Arise AI, OpenAI Agents SDK, Arize Skills

Source: https://www.youtube.com/watch?v=fnuNVFgLjHA