# Accelerate agent improvement with LangSmith Engine

## Executive summary

LangSmith Engine automates agent improvement by analyzing large volumes of production traces (e.g., 20,000 traces). It identifies recurring issues—such as agents sending outreach to disqualified prospects without warning—and proactively proposes code changes and updated instructions. Furthermore, it automatically generates evaluation datasets from failing runs, allowing engineers to test proposed fixes against historical failures before deployment.

## Key takeaways

- Automated Issue Identification: Engine reviews production traces to surface recurring problems (e.g., agent outreach despite CRM flagging a prospect as disqualified), grouping related issues across multiple agent runs.
- Proposed Fixes and Instructions: Beyond identifying the error, Engine drafts concrete changes: modifying tools (e.g., adding a 'confirmed disqualified' flag) and refining system prompts to include exceptions (e.g., requiring explicit confirmation before sending).
- Automated Evaluation Dataset Creation: Engine handles the tedious process of growing evaluation datasets by formatting failing production runs into examples, including reference outputs detailing how the agent should have behaved.

## Technical details

- Agent Monitoring and Debugging: The system can pinpoint exactly where an issue occurred within a trace (e.g., showing that `get prospects` tool call returned `disqualified is true`, but the agent still emailed).
- Code Integration Workflow: Proposed fixes can be quickly implemented via a Pull Request (PR) or pulled directly into a team's coding agent from LangSmith.
- Regression Monitoring: After merging a fix, Engine continues to track occurrences. If the same problem arises, the issue is reopened and resurfaced with an adjusted fix proposal.

## Practical implications

- Reduces manual effort required to sift through millions of agent traces in production.
- Accelerates the development cycle by providing actionable, pre-drafted fixes and instructions.
- Ensures continuous reliability by automatically monitoring for regressions after deployment.

## Topics

AI Agents, LLM Operations (LLMOps), Agent Reliability, Prompt Engineering, Data Observability, LangSmith Engine

Source: https://www.youtube.com/watch?v=hYhTke2m0X8