# Inside Clay's Eval Stack: 300M Agent Runs, One LangSmith Pipeline

## Executive summary

Clay detailed its agentic stack, featuring Claygent (go-to-market research) and Sculptor (go-to-market engineering workflow orchestration). The discussion emphasized that scaling agents to millions of runs requires a robust evaluation philosophy. Key architectural shifts include adopting a data lake for unified first/third-party data access, implementing comprehensive multi-stage evaluations (from local dev to CI), and leveraging large context models (like Fable) to enable complex, self-iterating agent loops.

## Key takeaways

- Agentic Systems are Becoming Interfaces: Clay is evolving into an 'agent interface,' making all parts of the product available via CLI and public API. This creates a flywheel where internal and external agents share the same tools, allowing failures in tool invocation to provide user signal for continuous improvement.
- Evals are Non-Negotiable for Scale: With billions of runs, comprehensive evaluation suites are critical. Clay's philosophy mandates different levels of testing: low-lift local dev checks, and staging/CI environments that mirror production harnesses.
- Data Lake Foundation for Agents: To handle disparate data sources (LangChain traces, Snowflake, Postgres, ClickHouse), Clay is moving to a data lake architecture. This allows agents to run wild on unified first-party and third-party data while enabling safe shadow builds.

## Technical details

- Agent Capabilities: Claygent is a research agent focused on web research and first-party data integration, running over 300 million runs monthly. Sculptor is the engineering agent used for building and orchestrating go-to-market workflows, handling over 100,000 messages weekly.
- Evaluation Philosophy (The Four Quadrants): Clay's eval coverage moves beyond simple 'goldens.' They utilize structured checks for complex queries and employ multi-turn evals. The most useful finding was that deterministic, multi-turn evals were more reliable than managing a noisy agent acting as the user.
- Addressing Evaluation Drift: The biggest challenge is 'eval drift' (production vs. test data). Clay mitigates this by incorporating customer support tickets, human-annotated goldens for judging drift, and using use case classifiers to ensure evals cover real production scenarios.
- Large Context Breakthrough: The availability of large context windows (e.g., Fable) allows agents to process massive datasets (e.g., 'Look at these 10,000 examples and find trends') in a single pass, moving beyond previous vibe-based analysis.

## Practical implications

- Build engineers should treat agent development as a continuous loop, where production failures (user signal) directly inform and improve offline evaluation suites.
- Adopting a data lake architecture is crucial for building scalable agents that need access to disparate first-party and third-party data sources.
- Designing APIs and CLIs with the explicit goal of being consumed by external agents ensures tool consistency and maximizes the development flywheel effect.

## Topics

Agentic AI, Evaluation Engineering (Evals), Data Architecture, CI/CD for LLMs, LangChain, LangSmith

Source: https://www.youtube.com/watch?v=Uny6LpmjraI