The Video Signal

Topic

Production Tracing

All digests tagged Production Tracing

Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI thumbnail

· 3:36

Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI

This walkthrough demonstrates a scalable workflow for analyzing high volumes of production traces by leveraging the LangSmith CLI and a coding agent. The process uses a 'perceived error' evaluator to flag potential failures, which are then systematically classified into defined issue types (e.g., flawed plan, context explosion). The final output is a structured, native thread dataset in LangSmith, organized into separate dataset splits for each issue type, making the failures readily available for evaluation metrics or post-training data.

Key takeaways

  1. Automated Failure Identification

    Use the 'perceived error' evaluator to flag threads containing evidence of agent mistakes or misunderstandings, transforming manual review into a data-driven process.

  2. Systematic Failure Classification 2:04

    Define specific agent issue types (e.g., Agent looping, flawed plan) and use a coding agent to classify hundreds of flagged threads into these categories.

  3. Building Structured Datasets

    The workflow culminates in creating a native thread dataset in LangSmith, ensuring a dedicated dataset split for every identified issue type. This provides a sorted, reusable corpus of failures.

Watch on YouTube Full article

Introducing: LangSmith Tuned Evaluators thumbnail

· 4:11

Introducing: LangSmith Tuned Evaluators

LangSmith Tuned Evaluators provide an automated, cost-effective way to attach quality feedback (signals) directly to production traces and threads for AI agents. These out-of-the-box evaluators analyze agent interactions—such as identifying perceived errors or misunderstandings—and surface failure modes that traditional system error logging misses. LangChain manages the entire evaluation pipeline, including prompt writing, judge model management, and inference infrastructure, allowing teams to focus on agent improvement workflows.

Key takeaways

  1. Automated Quality Feedback

    Tuned Evaluators automatically attach useful feedback signals to production traces and threads, helping identify agent behavior that needs attention (e.g., misunderstood user intent or contradictory answers).

  2. Perceived Error Detection

    The initial evaluator, Perceived Error, analyzes multi-turn conversations to detect potential mistakes by the agent, even when no explicit system error occurs. This signal can be derived from subtle patterns like unresolved outcomes or user pivots.

  3. Turnkey Management

    LangChain handles the entire evaluation lifecycle end-to-end: writing/testing prompts, managing judge models, benchmarking, and running inference infrastructure, eliminating the need for users to manage complex components. (See 0:28)

  4. Implementation Steps 0:12

    To use Tuned Evaluators, an organization admin must first enable the feature in LangSmith settings. After enabling, the evaluator can be attached to specific tracing projects.

Watch on YouTube Full article