Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic

Published 2026-10-12 · Duration 18:58

Summary

This workshop segment details advanced architectural patterns for deploying and managing Claude Managed Agents in enterprise environments. Key topics covered include integrating agent configurations into CI/CD pipelines using tools like `Ancho CLI`, achieving robust tenant isolation via workspaces, and implementing complex agent behaviors like the coordinator pattern for custom tools. The discussion emphasizes best practices for evaluation (Evals) and scaling compute resources using self-hosted sandboxes.

Download summary

Key takeaways

CI/CD for Agent Configurations
Use `Ancho CLI` to wrap the API and manage agent resources (defined in YAML) in a Terraform-like manner. This allows durable, long-term resources to be version-controlled and updated via CI/CD pipelines. (0:00)
Handling Custom Tool Waits
When an agent is waiting for a custom tool result, external messages are blocked (resulting in a 400 error). To maintain continuous interaction, implement a coordinator pattern using a sub-agent to handle tool execution while the main agent remains free to receive messages. (16:30, 18:30)
Data Quality for Evals 4:05
The best practice for evaluation harnesses is to focus on gathering high-quality, well-defined data. While internal tooling can spin up thousands of parallel sessions for testing, the quality of the input data dictates the performance of the model. (2:45)
Resource Isolation and Multi-Tenancy 14:08
For true data segregation (e.g., legal requirements), agents should be deployed in individual workspaces or an organization per customer. Resources are bound to the workspace, and the platform recommends using a metadata field (like `customer ID`) to map sessions to specific customers within a single workspace. (8:48, 9:54)
Scaling Compute Resources
For custom resource allocations (e.g., higher memory or dedicated GPUs), utilize self-hosted sandboxes. This allows integration with external providers like Modal, which offers GPU capabilities, rather than relying solely on the platform's provided sandboxes. (12:10)

Technical details

Agent Configuration Management 0s
Agent definitions and long-term resources should be managed using YAML files and synchronized via a CLI tool (e.g., `Ancho CLI`) within a CI/CD workflow, treating resources as code. (0:00)
Coordinator Pattern
To prevent the main agent from idling while a custom tool runs, a sub-agent can be designated to handle the tool execution. The main coordinator agent remains active, allowing continuous interaction and receiving messages, even if the sub-agent is processing the tool result. (18:30)
Multi-Agent Communication
A sub-agent can be used to process custom tools, emitting the tool use event. The main coordinator agent interacts with the sub-agent through the same core session thread, which correctly delegates the tool result back to the main thread. (18:30)
Long-Running Sessions
Long-running tasks are best managed by defining a clear exit criteria. The system will continue iterating until this criteria is met. Progress monitoring can be achieved by having a custom tool that the agent calls periodically to update the status. (14:34)

Mentioned resources

Ancho CLI (CLI Tool)
YAML (Configuration Format)
Modal (Compute Provider)
Claude Managed Agents docs (Documentation)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.