Inside Clay's Eval Stack: 300M Agent Runs, One LangSmith Pipeline
Clay detailed its agentic stack, featuring Claygent (go-to-market research) and Sculptor (go-to-market engineering workflow orchestration). The discussion emphasized that scaling agents to millions of runs requires a robust evaluation philosophy. Key architectural shifts include adopting a data lake for unified first/third-party data access, implementing comprehensive multi-stage evaluations (from local dev to CI), and leveraging large context models (like Fable) to enable complex, self-iterating agent loops.
Key takeaways
-
Agentic Systems are Becoming Interfaces
Clay is evolving into an 'agent interface,' making all parts of the product available via CLI and public API. This creates a flywheel where internal and external agents share the same tools, allowing failures in tool invocation to provide user signal for continuous improvement.
-
Evals are Non-Negotiable for Scale
3:23
With billions of runs, comprehensive evaluation suites are critical. Clay's philosophy mandates different levels of testing: low-lift local dev checks, and staging/CI environments that mirror production harnesses.
-
Data Lake Foundation for Agents
To handle disparate data sources (LangChain traces, Snowflake, Postgres, ClickHouse), Clay is moving to a data lake architecture. This allows agents to run wild on unified first-party and third-party data while enabling safe shadow builds.