We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI
Summary
Fuad Ali details the two-year evolution of Arize's in-product AI agent, Alex, covering architectural decisions from initial prototypes (50% hallucination) to a robust system. Key lessons for build engineers include moving planning logic into custom harness code rather than relying solely on the model, enforcing explicit turn-ending gates, treating large tool results as files (rather than dumping them into context), and opting for a small, capped markdown file for long-term memory over complex graph databases. The talk emphasizes that production sessions should be used as living test fixtures and that human judgment remains critical for quality vs. speed decisions.
Key takeaways
-
Harness Code Over Frameworks
5:35
The core architecture of Alex relies heavily on custom harness code for the loop, plan, stopping rule, and memory, providing maximum control and stability, rather than depending on a proprietary agent framework. This approach allows for robust engineering solutions even as frontier models improve.
-
Explicit Planning and Turn Control
9:00
The planning process was moved out of the model and into the harness code. A critical architectural decision was making 'finish' an explicit, enforced gate, preventing the model from losing focus or continuing unnecessarily.
-
Memory: Small File Beats Knowledge Graph
For long-term memory, the team found that a simple, capped markdown file (8,000 characters) written by a background job, loaded on every full request, was superior to complex systems like vector stores, episodic stores, or temporal knowledge graphs because it is small, exact, and easy for humans to audit.
-
Production Sessions as Living Tests
Live production traces are treated as valuable regression data sets and 'living fixtures.' This allows the system to continuously gather data on new error cases and user queries, making the production environment a primary source for testing and iteration.
Technical details
-
Agent Architecture Evolution
100s
Alex evolved from a simple router/skills tool-calling loop (2023 prototype, 50% hallucination) to a multi-step agent with planning, memory, and on-demand skills. The architecture now uses Signal (a trace reading agent) to debug Alex itself.
-
Tool Result Handling
830s
To prevent large tool results (e.g., big JSON blobs from trace analysis) from blowing up the context window, the system now truncates the data and uses the agent's code-writing capabilities (like JQ) to find the full context when necessary.
-
Model Selection Strategy
1180s
The team advises against assuming the newest model is always best. They generally default to 'fast and good enough' and focus engineering effort on the harness code, while exploring auto-routing to choose models per specific task.
-
Data and Testing
The system supports using entire sessions as data sets, allowing specific turns to be replayed for testing. Arize Experiments are used throughout the process to facilitate A/B testing of harness changes.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.