300 Agents, 60 People: How We Secured Our Own AI Agents - Aiden Sochowski, Runlayer
Summary
Founding engineer Aiden Sochowski details Runlayer's internal process for securing and monitoring over 300 AI agents used by a 60-person company. The core strategy involves deploying a central gateway to log every session and tool call, identifying risks like prompt injection, data exfiltration, and over-permissioned bots. Beyond security, the talk highlights the need for monitoring poorly written MCP servers and controlling runaway costs caused by model switches.
Key takeaways
-
Centralized Visibility is Critical
Many companies lack visibility into their agents. Runlayer solved this by deploying a gateway to log every session and tool call, allowing them to map agents by topic and risk.
-
Agent Risk Assessment
0:02
Agents can accelerate risks by mixing different classes of data. It is crucial to scope agents down and define policies for both inbound and outbound data flows.
-
Advanced Security Measures
0:06
To prevent attacks (like data exfiltration or PR approval hijacking), implementing classifier models to scrutinize every tool call and ensure alignment with the agent's defined intent is necessary. This is complex and context-dependent.
-
Cost and Reliability Monitoring
0:12
Monitoring is needed for poorly written MCP servers (which can lead to token cost blowups from formatting errors) and for tracking model switches (e.g., switching from GPT-4 to Claude 3.5, causing costs to balloon over 100x).
Technical details
-
Agent Architecture & Monitoring
3s
The solution involves deploying a gateway to log all agent sessions and tool calls centrally. This allows for visualization of the agent's activity, including which MCP servers are touched and what actions are taken.
-
Data Exfiltration Prevention
9s
Security policies can be defined to block PII from leaving externally (e.g., using regex on customer names in emails) or to block access to specific data sources (like private calendars).
-
Agent Scope and Intent Drift
11s
To prevent a single agent from contaminating or harming others, the only reliable method found is to tightly scope how agents can write to systems, ideally limiting writes to specific databases within a SaaS app.
-
Model Cost Control
14s
Cost blowups can occur when models are switched (e.g., from GPT-4 to Claude 3.5) or when poorly written MCP servers repeatedly encounter formatting errors, leading to excessive token usage.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.