Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma
Summary
The presentation outlines a comprehensive, governed framework for transitioning AI agents from experimental prototypes to secure, production-grade enterprise systems. The architecture emphasizes building a robust surrounding infrastructure—including centralized gateways, identity management, and observability—rather than focusing solely on the agent logic. Key components include a centralized MCP Gateway for secure authentication and routing, LangGraph for complex workflow management, and specialized frameworks for PII filtering, state management, and context optimization to ensure accountability, security, and cost efficiency.
Key takeaways
-
Build the Framework, Not Just the Agent
3:47
It is crucial to build the entire enterprise framework (including memory, tracking/evaluation, and context optimization) from the start, rather than rushing to deploy an agent. This ensures consistency and centralized governance across the organization.
-
Implement a Centralized Gateway (MCP Gateway)
8:12
All agent-to-tool calls must pass through a centralized MCP Gateway. This gateway handles authentication, routing, and ensures that agents never have direct access to underlying endpoints, enforcing security and accountability.
-
Ensure Observability and Auditing
11:24
Continuous monitoring is mandatory. The system must capture operational logs, traces (Tempo), and metrics (Prometheus) to track latency, failures, and access behavior, providing a complete audit trail for every agent action.
-
Manage State and Context Efficiently
17:36
Implement dedicated frameworks for agent memory management (using threads and checkpoints) and context optimization (pruning, re-ranking, deduplication) to maintain durable state and drastically reduce token usage, thereby controlling costs.
Technical details
-
Agent Development Lifecycle (CI/CD)
320s
The process requires agents to be registered in a central system. Local development involves building the agent as a Docker container and testing it against local memory/DBs. Promotion requires a PR, which registers the agent, followed by a CI/CD pipeline that scans the image for vulnerabilities and deploys it to a Kubernetes cluster using a dedicated orchestrator platform.
-
Agent Workflow (LangGraph)
541s
The agent uses a LangGraph framework to manage complex, multi-step workflows. The process involves interpreting the query, classifying the intent (read, modify, delete), executing the plan (calling the MCP Gateway), checking policies, and finally, executing the action or initiating a Human-in-the-Loop (HITL) approval process for sensitive operations.
-
Security and Data Filtering
1104s
A PII filtering layer is implemented to intercept and redact sensitive inputs (like API keys or documents) immediately upon ingestion, ensuring that secrets never reach the LLM, memory DB, or tracing systems.
-
Observability Stack
684s
The monitoring stack utilizes OpenTelemetry for collecting traces and metrics. Tempo stores detailed traces (e.g., which tool was slow, error details), while Prometheus tracks metrics (e.g., P95 latencies, authentication rejection spikes). Grafana aggregates this data for visualization.
Mentioned resources
- LangGraph
- MCP Gateway
- OpenTelemetry
- Tempo
- Prometheus
- Grafana
- Kubernetes
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.