# Introducing: LangSmith Tuned Evaluators

## Executive summary

LangSmith Tuned Evaluators provide an automated, cost-effective way to attach quality feedback (signals) directly to production traces and threads for AI agents. These out-of-the-box evaluators analyze agent interactions—such as identifying perceived errors or misunderstandings—and surface failure modes that traditional system error logging misses. LangChain manages the entire evaluation pipeline, including prompt writing, judge model management, and inference infrastructure, allowing teams to focus on agent improvement workflows.

## Key takeaways

- Automated Quality Feedback: Tuned Evaluators automatically attach useful feedback signals to production traces and threads, helping identify agent behavior that needs attention (e.g., misunderstood user intent or contradictory answers).
- Perceived Error Detection: The initial evaluator, Perceived Error, analyzes multi-turn conversations to detect potential mistakes by the agent, even when no explicit system error occurs. This signal can be derived from subtle patterns like unresolved outcomes or user pivots.
- Turnkey Management: LangChain handles the entire evaluation lifecycle end-to-end: writing/testing prompts, managing judge models, benchmarking, and running inference infrastructure, eliminating the need for users to manage complex components. (See 0:28)
- Implementation Steps: To use Tuned Evaluators, an organization admin must first enable the feature in LangSmith settings. After enabling, the evaluator can be attached to specific tracing projects.

## Technical details

- Tuned Evaluator Functionality: Tuned evaluators are out-of-the-box tools that attach useful feedback (e.g., Perceived Error) to production traces and threads, providing a scalable way to surface conversations for review.
- Perceived Error Logic: The Perceived Error evaluator looks for instances where an agent may have made a mistake or misunderstood the user, detecting signals from both explicit evidence (user correction) and subtle patterns (contradictory answers).
- Configuration & Deployment: After enabling the feature via LangSmith settings, users can attach the evaluator to a tracing project. Configuration options include changing the feedback tag name, adding filters, and setting the sampling rate.

## Practical implications

- Build engineers can integrate automated quality gates into their agent testing pipelines by using Tuned Evaluators to surface failure modes that are not captured by standard system error logs.
- The ability to use a specialized model (Perceived Error) that outperforms frontier models while reducing evaluation cost by up to 82% offers significant efficiency gains in AI product development.
- By managing the entire evaluator stack, LangChain reduces operational overhead for teams building and maintaining complex LLM-based agents.

## Topics

AI Agents, LLMOps, LangSmith, Evaluation Metrics, Production Tracing, Perceived Error Evaluator

Source: https://www.youtube.com/watch?v=0oh0X2NQgQo