# Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

## Executive summary

The presentation outlines a comprehensive, governed framework for transitioning AI agents from experimental prototypes to secure, production-grade enterprise systems. The architecture emphasizes building a robust surrounding infrastructure—including centralized gateways, identity management, and observability—rather than focusing solely on the agent logic. Key components include a centralized MCP Gateway for secure authentication and routing, LangGraph for complex workflow management, and specialized frameworks for PII filtering, state management, and context optimization to ensure accountability, security, and cost efficiency.

## Key takeaways

- Build the Framework, Not Just the Agent: It is crucial to build the entire enterprise framework (including memory, tracking/evaluation, and context optimization) from the start, rather than rushing to deploy an agent. This ensures consistency and centralized governance across the organization.
- Implement a Centralized Gateway (MCP Gateway): All agent-to-tool calls must pass through a centralized MCP Gateway. This gateway handles authentication, routing, and ensures that agents never have direct access to underlying endpoints, enforcing security and accountability.
- Ensure Observability and Auditing: Continuous monitoring is mandatory. The system must capture operational logs, traces (Tempo), and metrics (Prometheus) to track latency, failures, and access behavior, providing a complete audit trail for every agent action.
- Manage State and Context Efficiently: Implement dedicated frameworks for agent memory management (using threads and checkpoints) and context optimization (pruning, re-ranking, deduplication) to maintain durable state and drastically reduce token usage, thereby controlling costs.

## Technical details

- Agent Development Lifecycle (CI/CD): The process requires agents to be registered in a central system. Local development involves building the agent as a Docker container and testing it against local memory/DBs. Promotion requires a PR, which registers the agent, followed by a CI/CD pipeline that scans the image for vulnerabilities and deploys it to a Kubernetes cluster using a dedicated orchestrator platform.
- Agent Workflow (LangGraph): The agent uses a LangGraph framework to manage complex, multi-step workflows. The process involves interpreting the query, classifying the intent (read, modify, delete), executing the plan (calling the MCP Gateway), checking policies, and finally, executing the action or initiating a Human-in-the-Loop (HITL) approval process for sensitive operations.
- Security and Data Filtering: A PII filtering layer is implemented to intercept and redact sensitive inputs (like API keys or documents) immediately upon ingestion, ensuring that secrets never reach the LLM, memory DB, or tracing systems.
- Observability Stack: The monitoring stack utilizes OpenTelemetry for collecting traces and metrics. Tempo stores detailed traces (e.g., which tool was slow, error details), while Prometheus tracks metrics (e.g., P95 latencies, authentication rejection spikes). Grafana aggregates this data for visualization.

## Practical implications

- Adopt a centralized agent registry and dev portal to enforce consistent development standards and resource provisioning (memory, tracking, PII blocking) across all teams.
- Mandate that all tool calls, regardless of origin (local or agent), pass through a secure, authenticated MCP Gateway.
- Integrate comprehensive monitoring (OpenTelemetry, Prometheus) into the core architecture to ensure continuous visibility into agent performance, failures, and resource usage.
- Implement context optimization techniques (pruning, re-ranking, compression) to manage token costs and improve query efficiency.

## Topics

AI Agents, Enterprise Architecture, CI/CD, Observability, LLM Governance, Security Engineering, LangGraph, MCP Gateway, OpenTelemetry, Tempo, Prometheus, Grafana, Kubernetes

Source: https://www.youtube.com/watch?v=tZycGA_Fbfo