# How to go from your agent's traces to a fine-tuned model in one workflow

## Executive summary

This walkthrough introduces LangSmith Fine-Tuning and the SmithTune CLI, providing a single, end-to-end workflow to convert agent traces (trajectories) into custom, fine-tuned models. The process involves dataset creation and preparation from LangSmith trajectories, training using providers like Fireworks or Baseten, and subsequent evaluation, all managed through the SmithTune CLI. The overall workflow is: Create $\rightarrow$ Prepare $\rightarrow$ Plan $\rightarrow$ Train $\rightarrow$ Evaluate $\rightarrow$ Deploy.

## Key takeaways

- Supervised Fine-Tuning (SFT) Purpose: SFT teaches a model desired behavior (e.g., tool selection, workflow adherence) using 'golden examples' derived from agent traces, improving task accuracy and tool reliability.
- Trajectory Format Importance: The LangSmith trajectory format is critical because it provides a detailed reconstruction of the agent's context, including tools available at each turn, which is necessary for post-training jobs.
- SmithTune Workflow Overview: The process starts by pulling trajectories from a LangSmith tracing project, defining a task-specific rubric, having a model council judge the traces, and then pushing the approved traces back to LangSmith to form a training dataset.
- Training and Evaluation: After preparing the dataset (specifying provider, base model, and splits), the user runs `smithtune plan` to review settings, followed by `smithtune train` to execute the job. Evaluation compares the fine-tuned model against the base model on held-out test data.
- Deployment: If results meet quality targets, `smithtune deploy` is used to serve the tuned model and connect it to the application. Otherwise, the process iterates by refining examples or training settings.

## Technical details

- SmithTune CLI Usage: The CLI is used for the entire workflow: pulling trajectories, defining rubrics, pushing datasets, preparing splits, planning, training, and deploying. Key commands include `smithtune pull`, `smithtune dataset push`, `smithtune prepare`, `smithtune plan`, `smithtune train`, and `smithtune deploy`.
- Dataset Preparation: The dataset must be prepared by specifying the training provider (e.g., Baseten or Fireworks), the base model (e.g., Qwen 3.8 27B), and creating distinct training, validation, and test splits to prevent data leakage.
- Training Parameters: The `smithtune plan` command allows review of key configurations before training, including the selected model, number of examples, learning rate, batch size, and epochs.
- Model Comparison: After training, the results are published to LangSmith, providing a comparison view showing the base model vs. the fine-tuned model on held-out trajectories, including individual replay predictions and judge explanations.

## Practical implications

- Automates the complex MLOps pipeline for LLM agents, reducing manual steps required to move from raw agent logs to production-ready models.
- Provides a structured, reproducible method for data curation (using rubrics and model councils) before training, ensuring the 'golden dataset' is high quality.
- Allows engineers to systematically compare the performance gains of a fine-tuned model against a base model using held-out test data.

## Topics

Large Language Models (LLMs), Supervised Fine-Tuning (SFT), MLOps, Agentic Workflow, LangChain, CLI Development, SmithTune on GitHub, LangSmith, Open SWE

Source: https://www.youtube.com/watch?v=pb-AAvvQZ-U