AI ROI, Why the Agent Isn't the Answer
Summary
Achieving Return on Investment (ROI) with AI agents requires moving beyond simply implementing agents and instead focusing on the surrounding system architecture. Key constraints include the quality of the data context (what the agent can see) and the ability to measure performance (how to prove it's working). Speakers emphasized that the most valuable investments are in creating robust feedback loops, formal verification, and building federated, comprehensive data layers that preserve optionality and scope.
Key takeaways
-
Focus on the Feedback Loop, Not Just the Agent
20:00
The greatest ROI comes from investing in the feedback loop—the ability to automatically ingest failure modes and allow the system to improve itself. This is more critical than optimizing the agent itself.
-
Measure Full System Cost, Not Just Token Cost
26:40
To accurately measure ROI, the cost model must include all factors: retries, human attention, review cycles, and infrastructure, not just the token expenditure. Analyzing the full system cost allows for better optimization decisions.
-
Data Context is the Primary Constraint
38:20
Agents are limited by the data they receive. The solution is to build a federated, logical view of data that stitches together multiple sources (e.g., real-time Kafka data and long-term Iceberg data) to preserve optionality and scope.
-
Formal Verification for Stability
21:40
For mission-critical components, formal modeling (e.g., using TLA) is necessary to verify system behavior and prevent bugs, even when agents are constantly modifying the code base.
Technical details
-
Agentic Workflow Architecture
1100s
A robust agentic workflow involves five stages: defining a useful task, establishing shared context, agent execution (in a dev sandbox), verification/delivery, and continuous feedback loop improvement.
-
Data Blind Spots in AI
2000s
Common data blind spots include outdated data, wrong scoping, missing relationships between departments/products, and the inability to distinguish between 'no data' and 'unknown state'.
-
Data Federation and Logical Views
2300s
Instead of relying on single data sources, the best practice is to create a logical view by federating across multiple systems (e.g., combining real-time data from Apache Kafka with historical data in Apache Iceberg) to provide a comprehensive context.
-
Formal Modeling
1300s
Formal methods (like TLA) can be used to model and verify complex system behaviors (e.g., a work queue/Q) to ensure stability and prevent bugs introduced by autonomous agents.
Mentioned resources
- Anthropic
- OpenAI
- Google Cloud
- TLA (Temporal Logic of Actions)
- Apache Kafka
- Apache Iceberg
- Open Metadata
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.