Accelerate agent improvement with LangSmith Engine
LangSmith Engine automates agent improvement by analyzing large volumes of production traces (e.g., 20,000 traces). It identifies recurring issues—such as agents sending outreach to disqualified prospects without warning—and proactively proposes code changes and updated instructions. Furthermore, it automatically generates evaluation datasets from failing runs, allowing engineers to test proposed fixes against historical failures before deployment.
Key takeaways
-
Automated Issue Identification
Engine reviews production traces to surface recurring problems (e.g., agent outreach despite CRM flagging a prospect as disqualified), grouping related issues across multiple agent runs.
-
Proposed Fixes and Instructions
2:05
Beyond identifying the error, Engine drafts concrete changes: modifying tools (e.g., adding a 'confirmed disqualified' flag) and refining system prompts to include exceptions (e.g., requiring explicit confirmation before sending).
-
Automated Evaluation Dataset Creation
2:40
Engine handles the tedious process of growing evaluation datasets by formatting failing production runs into examples, including reference outputs detailing how the agent should have behaved.