Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI
This walkthrough demonstrates a scalable workflow for analyzing high volumes of production traces by leveraging the LangSmith CLI and a coding agent. The process uses a 'perceived error' evaluator to flag potential failures, which are then systematically classified into defined issue types (e.g., flawed plan, context explosion). The final output is a structured, native thread dataset in LangSmith, organized into separate dataset splits for each issue type, making the failures readily available for evaluation metrics or post-training data.
Key takeaways
-
Automated Failure Identification
Use the 'perceived error' evaluator to flag threads containing evidence of agent mistakes or misunderstandings, transforming manual review into a data-driven process.
-
Systematic Failure Classification
2:04
Define specific agent issue types (e.g., Agent looping, flawed plan) and use a coding agent to classify hundreds of flagged threads into these categories.
-
Building Structured Datasets
The workflow culminates in creating a native thread dataset in LangSmith, ensuring a dedicated dataset split for every identified issue type. This provides a sorted, reusable corpus of failures.