How to go from your agent's traces to a fine-tuned model in one workflow
Summary
This walkthrough introduces LangSmith Fine-Tuning and the SmithTune CLI, providing a single, end-to-end workflow to convert agent traces (trajectories) into custom, fine-tuned models. The process involves dataset creation and preparation from LangSmith trajectories, training using providers like Fireworks or Baseten, and subsequent evaluation, all managed through the SmithTune CLI. The overall workflow is: Create $\rightarrow$ Prepare $\rightarrow$ Plan $\rightarrow$ Train $\rightarrow$ Evaluate $\rightarrow$ Deploy.
Key takeaways
-
Supervised Fine-Tuning (SFT) Purpose
SFT teaches a model desired behavior (e.g., tool selection, workflow adherence) using 'golden examples' derived from agent traces, improving task accuracy and tool reliability.
-
Trajectory Format Importance
0:01
The LangSmith trajectory format is critical because it provides a detailed reconstruction of the agent's context, including tools available at each turn, which is necessary for post-training jobs.
-
SmithTune Workflow Overview
0:02
The process starts by pulling trajectories from a LangSmith tracing project, defining a task-specific rubric, having a model council judge the traces, and then pushing the approved traces back to LangSmith to form a training dataset.
-
Training and Evaluation
0:07
After preparing the dataset (specifying provider, base model, and splits), the user runs `smithtune plan` to review settings, followed by `smithtune train` to execute the job. Evaluation compares the fine-tuned model against the base model on held-out test data.
-
Deployment
0:08
If results meet quality targets, `smithtune deploy` is used to serve the tuned model and connect it to the application. Otherwise, the process iterates by refining examples or training settings.
Technical details
-
SmithTune CLI Usage
3s
The CLI is used for the entire workflow: pulling trajectories, defining rubrics, pushing datasets, preparing splits, planning, training, and deploying. Key commands include `smithtune pull`, `smithtune dataset push`, `smithtune prepare`, `smithtune plan`, `smithtune train`, and `smithtune deploy`.
-
Dataset Preparation
5s
The dataset must be prepared by specifying the training provider (e.g., Baseten or Fireworks), the base model (e.g., Qwen 3.8 27B), and creating distinct training, validation, and test splits to prevent data leakage.
-
Training Parameters
6s
The `smithtune plan` command allows review of key configurations before training, including the selected model, number of examples, learning rate, batch size, and epochs.
-
Model Comparison
7s
After training, the results are published to LangSmith, providing a comparison view showing the base model vs. the fine-tuned model on held-out trajectories, including individual replay predictions and judge explanations.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.