# Agents Aren't Stupid, They're Just Blind

## Executive summary

AI agents are not inherently flawed, but they are 'blind' due to poor data context. The solution is not building smarter models, but implementing robust context engineering at the data layer. This involves moving away from designing data for single business problems (1:1 mapping) and instead building a unified, federated data context that preserves optionality. Key architectural patterns include stitching real-time data (like Apache Kafka) and long-term storage (like Apache Iceberg) into a single logical view, ensuring temporal consistency and comprehensive governance.

## Key takeaways

- The Problem of Blind Agents: Agents can provide confidently wrong answers by relying on outdated, inaccurate, or incorrectly scoped data, leading to significant data loss even if human interaction costs are saved. (00:06:00)
- Data Access Limitations: Current methods—RAG (best for static corpora), MCP (good for fresh, wide-scope actions), and Skills (not designed for direct data access)—each have limitations that prevent a complete, accurate view of reality. (00:10:02)
- The Solution: Context Engineering: Instead of building for a single business problem, data architecture must be designed for 'context.' This requires federation and logical views to preserve optionality and allow agents to work with a broader, consistent data scope. (00:20:15)
- Achieving Temporal Consistency: A critical implementation pattern is stitching real-time systems (e.g., Apache Kafka) with long-term storage (e.g., Apache Iceberg) into a single logical view, ensuring data consistency across time spans. (00:26:18)

## Technical details

- Data Blind Spots: Four common failure modes include: 1) Using outdated/inaccurate data; 2) Wrong/over-scoping data; 3) Missing relationships between departments/products; and 4) Confidently answering when data is unknown. (00:06:00)
- Data Architectures: The traditional Medallion Architecture (Bronze/Silver/Gold) is effective for 1:1 problem solutions but struggles with the complexity of agentic AI. (00:12:48)
- Data Access Patterns: RAG is suitable for large, static corpora; MCP is useful for accessing fresh, wide-scope data via APIs; Skills are not designed for direct data access. (00:10:02)
- Federation and Logical Views: Federation allows issuing a single query that maps to multiple data sets and systems, delaying materialization and pushing processing as far left in the architecture as possible. (00:20:15)
- Data Governance: A unified governance context and overarching catalog (like OpenMetadata) are necessary to apply policies across federated data sources. (00:32:55)

## Practical implications

- Shift the focus from building for a specific business problem to building for a consistent, broad 'context.'
- Implement data federation and logical views to unify data across disparate systems (e.g., Kafka and Iceberg).
- Prioritize designing data interfaces that preserve optionality and can answer multiple, varied questions.
- Utilize an overarching catalog layer to enforce governance and policies across the federated view.

## Topics

AI Agents, Data Engineering, Context Engineering, Data Architecture, Federation, LLMs, Data Governance, Streambased, OpenMetadata, Apache Kafka, Apache Iceberg

Source: https://www.youtube.com/watch?v=LBo4eMkQW8c