# Inside the Agent Engine: A LangChain and Traversal Fireside Chat

## Executive summary

The discussion details the challenges and architectural requirements for building AI Site Reliability Engineering (SRE) agents capable of handling petabyte-scale production incidents. Speakers emphasize that SRE troubleshooting is uniquely difficult due to the lack of labeled data, high stakes, and massive telemetry volumes. Successful agent design requires moving beyond simple RAG/vector search by implementing sophisticated 'agent harnesses' that manage context via file systems, build a comprehensive 'production world model,' and strategically balance offline vs. online computation.

## Key takeaways

- SRE Agents Face Unique Data Challenges: Troubleshooting is difficult because there is no good labeled data for LLMs to train on, human troubleshooting processes are complex, and the scale of telemetry (e.g., petabytes per day) makes traditional context window methods infeasible.
- Agent Architecture Requires a Core/Sub-Agent Harness: Instead of monolithic agents, the recommended approach is building one core agent that orchestrates multiple specialized sub-agents. This requires a robust harness to manage context and file systems effectively.
- The Production World Model is Key to System Knowledge: Learning system knowledge involves synthesizing large streams of non-telemetry data (e.g., code, Slack) with raw observability logs to build a 'production world model,' which acts as the system's deep wiki.
- Evaluation Must Focus on Hardest Tasks: When evaluating agents, focus on the hardest tasks (like incident RCA) because success in these complex areas tends to generalize better than focusing on easier, less representative tasks.

## Technical details

- Data Scale and Search Strategy: At petabyte scale, agents are bottlenecked by raw API calls. Effective search requires re-indexing and owning components to make data more searchable. The strategy must straddle the continuum between high-granularity, live online queries (e.g., Datadog) and lower-granularity, massive offline computation.
- Agent Maturity Levels (L0 to L5): The complexity of building agents follows a framework similar to self-driving cars: L0 is manual troubleshooting; L1 uses runbooks/rules; L2 involves LLMs assisting with painful tasks (e.g., writing Splunk queries); L3 works on smaller, specific microservices; and L4 requires thinking across the entire petabyte production environment.
- Memory Differentiation: The 'production world model' represents synthesized system knowledge (telemetry, code). This is distinct from a 'knowledge bank,' which stores user-provided documentation and personal preferences/taste. User memory should be kept separate to prevent false tribal knowledge from corrupting the system fidelity.
- Agent Autonomy Progression: The product direction is moving from manually triggered agents $\rightarrow$ rule-based systems $\rightarrow$ chatbot-style background agents $\rightarrow$ proactive agents with inherent agency.

## Practical implications

- When building agents for complex systems, prioritize developing a robust 'harness' layer that manages context and file system interactions, rather than relying solely on the LLM's core capabilities.
- To manage cost and performance at scale, implement advanced controls (like an LLM gateway) to govern token usage and allocate resources based on user/task criticality.
- Start development with a simple single-agent loop and skills before attempting complex multi-agent swarms; this approach often yields better results for early implementations.

## Topics

AI SRE, Agent Architecture, Observability, Large Language Models (LLMs), System Reliability Engineering (SRE), LangChain, Traversal (Company), Deep Agents

Source: https://www.youtube.com/watch?v=U5PkKt_uJys