Accelerate agent improvement with LangSmith Engine
Summary
LangSmith Engine automates agent improvement by analyzing large volumes of production traces (e.g., 20,000 traces). It identifies recurring issues—such as agents sending outreach to disqualified prospects without warning—and proactively proposes code changes and updated instructions. Furthermore, it automatically generates evaluation datasets from failing runs, allowing engineers to test proposed fixes against historical failures before deployment.
Key takeaways
-
Automated Issue Identification
Engine reviews production traces to surface recurring problems (e.g., agent outreach despite CRM flagging a prospect as disqualified), grouping related issues across multiple agent runs.
-
Proposed Fixes and Instructions
2:05
Beyond identifying the error, Engine drafts concrete changes: modifying tools (e.g., adding a 'confirmed disqualified' flag) and refining system prompts to include exceptions (e.g., requiring explicit confirmation before sending).
-
Automated Evaluation Dataset Creation
2:40
Engine handles the tedious process of growing evaluation datasets by formatting failing production runs into examples, including reference outputs detailing how the agent should have behaved.
Technical details
-
Agent Monitoring and Debugging
80s
The system can pinpoint exactly where an issue occurred within a trace (e.g., showing that `get prospects` tool call returned `disqualified is true`, but the agent still emailed).
-
Code Integration Workflow
135s
Proposed fixes can be quickly implemented via a Pull Request (PR) or pulled directly into a team's coding agent from LangSmith.
-
Regression Monitoring
190s
After merging a fix, Engine continues to track occurrences. If the same problem arises, the issue is reopened and resurfaced with an adjusted fix proposal.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.