Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic
This workshop segment details advanced architectural patterns for deploying and managing Claude Managed Agents in enterprise environments. Key topics covered include integrating agent configurations into CI/CD pipelines using tools like `Ancho CLI`, achieving robust tenant isolation via workspaces, and implementing complex agent behaviors like the coordinator pattern for custom tools. The discussion emphasizes best practices for evaluation (Evals) and scaling compute resources using self-hosted sandboxes.
Key takeaways
-
CI/CD for Agent Configurations
Use `Ancho CLI` to wrap the API and manage agent resources (defined in YAML) in a Terraform-like manner. This allows durable, long-term resources to be version-controlled and updated via CI/CD pipelines. (0:00)
-
Handling Custom Tool Waits
When an agent is waiting for a custom tool result, external messages are blocked (resulting in a 400 error). To maintain continuous interaction, implement a coordinator pattern using a sub-agent to handle tool execution while the main agent remains free to receive messages. (16:30, 18:30)
-
Data Quality for Evals
4:05
The best practice for evaluation harnesses is to focus on gathering high-quality, well-defined data. While internal tooling can spin up thousands of parallel sessions for testing, the quality of the input data dictates the performance of the model. (2:45)
-
Resource Isolation and Multi-Tenancy
14:08
For true data segregation (e.g., legal requirements), agents should be deployed in individual workspaces or an organization per customer. Resources are bound to the workspace, and the platform recommends using a metadata field (like `customer ID`) to map sessions to specific customers within a single workspace. (8:48, 9:54)
-
Scaling Compute Resources
For custom resource allocations (e.g., higher memory or dedicated GPUs), utilize self-hosted sandboxes. This allows integration with external providers like Modal, which offers GPU capabilities, rather than relying solely on the platform's provided sandboxes. (12:10)