What Does an AI Engineer Do Now? - Aparna Dhinakaran, Arize AI
Summary
The role of the AI Engineer is rapidly evolving from writing isolated features to building complex, self-improving 'loops' or 'factories of agents.' This new paradigm requires building systems that manage non-deterministic work, consume telemetry data, and continuously improve through evaluation. Key architectural components include always-on agents, sandboxed execution environments, and robust evaluation systems that test the agents' outputs against human judgment.
Key takeaways
-
The Shift from Features to Loops
2:23
The AI engineer's job is no longer just writing an AI feature; it is building the entire loop: setting up systems and agents for code reviews, writing e-vals for non-deterministic tasks, and creating agents that consume observability data to find patterns. (02:23)
-
Agents as Primary Telemetry Consumers
4:45
The future of observability data consumption is shifting from human review to agents. Telemetry data must be structured so that it can serve as a trigger for always-on agents to run and identify issues. (04:45)
-
The Importance of Evaluating Evaluators
10:21
A critical function for AI engineers is to evaluate the evaluators (e-vals). This involves building 'golden truth' datasets and verifying that the automated judge aligns with human judgment before deployment. (10:21)
-
Tracing the Entire Factory
12:48
The goal is to trace everything, including the coding agents (e.g., Cloud Code, Cursor, Codex) that build the system. This allows for understanding the entire 'factory of agents' and running online evaluations on the resulting telemetry. (12:48)
Technical details
-
Signal Architecture
211s
Signal is described as an always-on agent that reads traces and telemetry to identify patterns and can automatically generate fixes or Pull Requests (PRs). It operates by reading traces and identifying issues from telemetry data, while maintaining memory of previously seen issues. (03:31)
-
Agent Execution Environment
361s
An agent run requires pulling in context (APM logs, APM metrics, third-party telemetry), access to the repo, and keys. All this data is contained within a 'sandbox,' which is accessed only through a single 'proxy.' (06:01)
-
Durable Agents and State Management
493s
To ensure reliability, agents must be designed to be durable. State (like found issues or learned skills) is saved to external memory, allowing the agent to pick up from the point of failure rather than restarting from scratch. (08:13)
-
Agent Agnosticism
571s
The system is designed to be model and harness agnostic, supporting multiple models (e.g., Claude Code, Cursor, Codex) and aiming to support all major open-source harnesses. (09:14)
Mentioned resources
- Arize AI
- Signal
- AIE CODE summit
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.