Supervised Fine-Tuning (SFT)

SFT teaches a model desired behavior (e.g., tool selection, workflow adherence) using 'golden examples' derived from agent traces, improving task accuracy and tool reliability.

The LangSmith trajectory format is critical because it provides a detailed reconstruction of the agent's context, including tools available at each turn, which is necessary for post-training jobs.

The process starts by pulling trajectories from a LangSmith tracing project, defining a task-specific rubric, having a model council judge the traces, and then pushing the approved traces back to LangSmith to form a training dataset.

After preparing the dataset (specifying provider, base model, and splits), the user runs `smithtune plan` to review settings, followed by `smithtune train` to execute the job. Evaluation compares the fine-tuned model against the base model on held-out test data.

If results meet quality targets, `smithtune deploy` is used to serve the tuned model and connect it to the application. Otherwise, the process iterates by refining examples or training settings.

Post-training builds upon pre-training (which uses massive, diverse, unstructured data for causal language modeling) by focusing on structured data. This teaches the model to follow instructions, use chat templates, and emit tool calls, moving beyond simple task-specific or multi-task training.

To narrow model focus, post-training emphasizes structured data (e.g., tool responses, user prompts, tool sets) over unstructured text. This allows the model to learn specific formats, such as JSON or XML, for tool interaction.

To specialize a model (e.g., for software engineering) without losing general capabilities, techniques include including general chat data in the blend and using algorithmic approaches like multi-teacher on-policy distillation (MOPD).

Data quality is paramount. Researchers must analyze data statistics (e.g., trajectory length, tool call diversity) and perform ablation studies to identify and address biases (e.g., over-reliance on a single tool) that could cause model failure in real-world use.

For developers, the process involves defining the model's 'soul' (core capabilities), quantifying these goals via benchmarks, setting quantitative milestones, and iterating through the post-training pipeline. Starting resources include Nemotron 3.5 Lightning.

The standard self-attention mechanism has $O(T^2)$ complexity (where T is sequence length). To mitigate this, techniques like Grouped Query Attention (GQA) reduce the number of keys and values used across heads by mapping multiple query groups to a smaller set of shared keys/values. Similarly, Sliding Window Attention limits attention to only recent history, reducing complexity to $O(T imes W)$ where W is the window size.

MoE allows models to have a large total parameter count (e.g., 30B) while keeping the active computation small (e.g., 3B). This is achieved by using a routing module that directs an input vector to only a subset of specialized expert sub-networks, significantly improving compute efficiency.

For downstream tasks (e.g., sentiment analysis), models can be adapted using In-Context Learning (ICL). This involves concatenating task examples and the test input into the prompt sequence without updating model parameters, which is fundamentally different from traditional fine-tuning.

SFT involves collecting data in an instruction/answer pair format and training the model by minimizing the negative log likelihood of predicting the answer ($Y$) given the instruction ($X$). This is a supervised process that updates the model's weights.