# Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI

## Executive summary

This walkthrough demonstrates a scalable workflow for analyzing high volumes of production traces by leveraging the LangSmith CLI and a coding agent. The process uses a 'perceived error' evaluator to flag potential failures, which are then systematically classified into defined issue types (e.g., flawed plan, context explosion). The final output is a structured, native thread dataset in LangSmith, organized into separate dataset splits for each issue type, making the failures readily available for evaluation metrics or post-training data.

## Key takeaways

- Automated Failure Identification: Use the 'perceived error' evaluator to flag threads containing evidence of agent mistakes or misunderstandings, transforming manual review into a data-driven process.
- Systematic Failure Classification: Define specific agent issue types (e.g., Agent looping, flawed plan) and use a coding agent to classify hundreds of flagged threads into these categories.
- Building Structured Datasets: The workflow culminates in creating a native thread dataset in LangSmith, ensuring a dedicated dataset split for every identified issue type. This provides a sorted, reusable corpus of failures.

## Technical details

- LangSmith CLI Workflow: The process involves using the LangSmith CLI and a coding agent to interact with production traces. Steps include grabbing the latest LangSmith skills and pulling a defined number of flagged threads (e.g., 50 latest threads flagged for perceived error).
- Agent Classification: A defined list of agent issue types (e.g., Agent looping, context explosion, failed recovery, feature gap, flawed plan) is provided to the coding agent to classify the perceived error failures by reading both the thread and the feedback reasoning.
- Dataset Creation: The final step uses a prompt to instruct the coding agent to create a native thread dataset in LangSmith, ensuring a separate dataset split is generated for each defined issue type.

## Practical implications

- Allows engineering teams to efficiently analyze thousands of production interactions without manual review.
- Provides a structured, categorized corpus of real-world failures, which can be used as robust evaluation metrics or post-training examples for LLM agents.
- Enables targeted debugging by isolating failures into specific categories (e.g., 'flawed plan' failures).

## Topics

LangSmith, LLM Agents, Dataset Engineering, Production Tracing, Failure Analysis, LangSmith docs, LangSmith CLI and skills

Source: https://www.youtube.com/watch?v=hd-LmPwfrVc