AI ROI, Why the Agent Isn't the Answer
Achieving Return on Investment (ROI) with AI agents requires moving beyond simply implementing agents and instead focusing on the surrounding system architecture. Key constraints include the quality of the data context (what the agent can see) and the ability to measure performance (how to prove it's working). Speakers emphasized that the most valuable investments are in creating robust feedback loops, formal verification, and building federated, comprehensive data layers that preserve optionality and scope.
Key takeaways
-
Focus on the Feedback Loop, Not Just the Agent
20:00
The greatest ROI comes from investing in the feedback loop—the ability to automatically ingest failure modes and allow the system to improve itself. This is more critical than optimizing the agent itself.
-
Measure Full System Cost, Not Just Token Cost
26:40
To accurately measure ROI, the cost model must include all factors: retries, human attention, review cycles, and infrastructure, not just the token expenditure. Analyzing the full system cost allows for better optimization decisions.
-
Data Context is the Primary Constraint
38:20
Agents are limited by the data they receive. The solution is to build a federated, logical view of data that stitches together multiple sources (e.g., real-time Kafka data and long-term Iceberg data) to preserve optionality and scope.
-
Formal Verification for Stability
21:40
For mission-critical components, formal modeling (e.g., using TLA) is necessary to verify system behavior and prevent bugs, even when agents are constantly modifying the code base.