Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI

Summary

This walkthrough demonstrates a scalable workflow for analyzing high volumes of production traces by leveraging the LangSmith CLI and a coding agent. The process uses a 'perceived error' evaluator to flag potential failures, which are then systematically classified into defined issue types (e.g., flawed plan, context explosion). The final output is a structured, native thread dataset in LangSmith, organized into separate dataset splits for each issue type, making the failures readily available for evaluation metrics or post-training data.

Technical details

• LangSmith CLI Workflow 0s

  The process involves using the LangSmith CLI and a coding agent to interact with production traces. Steps include grabbing the latest LangSmith skills and pulling a defined number of flagged threads (e.g., 50 latest threads flagged for perceived error).

• Agent Classification 138s

  A defined list of agent issue types (e.g., Agent looping, context explosion, failed recovery, feature gap, flawed plan) is provided to the coding agent to classify the perceived error failures by reading both the thread and the feedback reasoning.

• Dataset Creation

  The final step uses a prompt to instruct the coding agent to create a native thread dataset in LangSmith, ensuring a separate dataset split is generated for each defined issue type.

Mentioned resources

• LangSmith (Platform)
• LangSmith docs (Documentation)
• LangSmith CLI and skills (Documentation)

Channel & topics

• LangChain
• LangSmith
• LLM Agents
• Dataset Engineering
• Production Tracing
• Failure Analysis
• LangSmith docs
• LangSmith CLI and skills