MLOps — The Video Signal

Sep 24, 2026 · 9:01

How to go from your agent's traces to a fine-tuned model in one workflow

This walkthrough introduces LangSmith Fine-Tuning and the SmithTune CLI, providing a single, end-to-end workflow to convert agent traces (trajectories) into custom, fine-tuned models. The process involves dataset creation and preparation from LangSmith trajectories, training using providers like Fireworks or Baseten, and subsequent evaluation, all managed through the SmithTune CLI. The overall workflow is: Create $\rightarrow$ Prepare $\rightarrow$ Plan $\rightarrow$ Train $\rightarrow$ Evaluate $\rightarrow$ Deploy.

Key takeaways

Supervised Fine-Tuning (SFT) Purpose
SFT teaches a model desired behavior (e.g., tool selection, workflow adherence) using 'golden examples' derived from agent traces, improving task accuracy and tool reliability.
Trajectory Format Importance 0:01
The LangSmith trajectory format is critical because it provides a detailed reconstruction of the agent's context, including tools available at each turn, which is necessary for post-training jobs.
SmithTune Workflow Overview 0:02
The process starts by pulling trajectories from a LangSmith tracing project, defining a task-specific rubric, having a model council judge the traces, and then pushing the approved traces back to LangSmith to form a training dataset.
Training and Evaluation 0:07
After preparing the dataset (specifying provider, base model, and splits), the user runs `smithtune plan` to review settings, followed by `smithtune train` to execute the job. Evaluation compares the fine-tuned model against the base model on held-out test data.
Deployment 0:08
If results meet quality targets, `smithtune deploy` is used to serve the tuned model and connect it to the application. Otherwise, the process iterates by refining examples or training settings.

Watch on YouTube Full article

Sep 22, 2026 · 5:33

What is LangSmith?

LangSmith is a comprehensive platform designed for the Agent Development Lifecycle (ADLC), enabling build engineers to build, test, deploy, and monitor LLM applications and agents. It functions as a tracing backend, providing crucial observability into complex agent behavior—which can involve dozens of model and tool calls—by tracking every step, diagnosing bugs, and facilitating continuous quality assurance through structured testing and production monitoring.

Key takeaways

Agent Observability is Critical
Agents are inherently 'black boxes'; LangSmith solves this by providing visibility into the sequence of model calls and tool decisions, which are not visible in the final output.
Tracing Components 0:01
LangSmith defines three components: a 'Run' (a single unit of work, e.g., one model call or tool call), a 'Trace' (a full pass through the agent, composed of multiple runs), and a 'Thread' (a conversation grouping multiple traces from one customer interaction).
Testing and Validation Loop 0:02
The platform uses Datasets (sets of examples), Evaluators (which score examples, potentially using an LLM-as-a-judge), and Experiments (running agents over datasets) to verify fixes and compare performance changes (regression testing).
Production Monitoring 0:03
In production, LangSmith allows online evaluators to score live traffic, generating dashboards that track scores, volume, latency, errors, and cost, and can trigger alerts or route traces to annotation queues.

Watch on YouTube Full article

Sep 22, 2026 · 14:35

100x More Reliable Voice Agents

Unmute is introduced as an open, declarative standard for building voice agents, designed to solve the critical problem of vendor lock-in and architectural fragility in the current voice AI landscape. Current agents are often embedded within specific platforms (e.g., ElevenLabs, VAPI, LiveKit), making them non-portable. Unmute shifts the paradigm by declaring the agent's behavior, tool calls, data flows, and failure handling in a single, compiled package, making the LLM an optional resource rather than the sole decision-making spine. This approach ensures structural guarantees and portability across different cloud runtimes.

Key takeaways

Declarative Execution and Portability 2:00
Unmute allows the agent's behavior to be defined in a single, portable package that compiles to various runtimes (e.g., PipeChat Cloud, LiveKit Cloud, or Slang's architecture). This means the agent's core logic remains version-controlled IP, regardless of the underlying infrastructure or cloud region.
Addressing LLM Over-Reliance 3:00
Traditional agents assume the LLM is the decision-making spine, leading to unpredictable, expensive, and difficult-to-govern behavior. Unmute enforces execution through declared steps, making the LLM an optional resource that must 'earn its place' in the call.
Structural Guarantees over Instructions 4:00
The standard moves beyond the false choice between rigid IVR and complex single-prompt agents. By splitting the work into discrete, context-limited steps, the system achieves determinism—allowing specific rules (like a regulated disclosure) to be locked down while other parts remain generative, all within the same call.

Watch on YouTube Full article

Sep 9, 2026 · 8:44

Accelerate the self-improving AI loop with CoreWeave ARIA

CoreWeave ARIA is an AI research and iteration agent integrated into Weights & Biases (W&B) designed to accelerate the self-improving AI loop. It addresses common challenges in AI development, such as stalled iteration cycles, massive data volume analysis, and manual dashboard creation. ARIA automates auto-research, analyzes training metrics and agent traces, generates comprehensive reports with suggested next steps, and assists in optimizing LLM prompts and agent performance.

Key takeaways

Automated Auto-Research Loop
ARIA can conduct auto-research by analyzing recorded training metrics and agent traces to uncover hidden insights. It generates visualization-packed W&B reports and automatically launches follow-up training experiments based on its findings, minimizing manual effort (5:51).
Agent Performance Optimization
ARIA supports agent development by analyzing production traces and suggesting improvements. It can specifically help refine system prompts and evaluate multiple prompt alternatives using defined datasets to achieve higher quality results at lower latency (7:07).
Comprehensive Workflow Support 2:30
Beyond research, ARIA handles time-consuming manual tasks like providing advice, generating code, and executing commands, all while supporting concurrent conversations that can continue running in the cloud (2:21).

Watch on YouTube Full article

Sep 8, 2026 · 54:15

Specializing AI for Regulated Industries - How Domyn Uses NVIDIA Nemotron

This livestream details Domyn's journey toward building a family of sovereign AI models for regulated industries, emphasizing full ownership and control over the entire model stack. The presentation covers advanced LLM development techniques—including model compression (pruning/distillation), continual pretraining (CPT), Supervised Fine-Tuning (SFT), and Reinforcement Learning (RL)—using the NVIDIA open source ecosystem. A significant focus is placed on the tooling required to manage these complex pipelines at scale, including custom tools like Swarm and FinalFlows, which are crucial for build engineers managing large-scale compute infrastructure.

Key takeaways

Sovereign AI requires full stack control 0:28
For regulated industries, achieving sovereign AI necessitates controlling every layer of the stack—from compute to model ownership—rather than relying on point solutions. This approach is critical for governance and auditability. (0:28-1:35)
Domain Large development pipeline 7:16
Domyn developed Domain Large by starting with Coliseum 355, followed by model compression (pruning/distillation), CPT to expand context up to 128K tokens, and SFT to enable reasoning. This was executed using the NVIDIA stack on H200s in DJX cloud. (4:36-7:39)
Domain Small for efficiency 23:43
To address cost concerns associated with large models, Domain Small (10B parameters) was created. Its training heavily leveraged Reinforcement Learning (RL) and Direct Preference Optimization (DPO), demonstrating that smaller models can achieve strong performance through advanced post-training curricula. (14:23-17:56)
Tooling for scalable ML pipelines 40:08
Domyn developed internal tools like Swarm (CLI/Python) and FinalFlows (DAG library) to manage complex, interconnected jobs on Slurm clusters. These tools remove friction when running large-scale evaluations and training across European infrastructure. (24:08-31:56)

Watch on YouTube Full article

Aug 17, 2026 · 56:10

Reading Group July 2026 - Loop Engineering

The session defines 'Loop Engineering' as a fundamental shift in AI development, moving beyond manual prompt-by-prompt interaction toward designing autonomous control systems. These loops automate complex software engineering tasks by having agents discover work, delegate sub-tasks, verify results, persist state, and self-optimize until a goal is met. Speakers detailed the evolution from simple prompts to sophisticated multi-agent architectures that aim to industrialize the entire software development lifecycle, emphasizing robust validation, evaluation layers, and continuous feedback mechanisms.

Key takeaways

The Evolution of AI Development 3:50
Software automation progressed through stages: Prompt Engineering $ ightarrow$ Context Engineering $ ightarrow$ Harness Engineering $ ightarrow$ Loop Engineering. The goal is to build 'software factories' that self-verify and optimize, rather than requiring manual verification after every turn.
Implementing Robust Loops 50:51
Building production loops requires more than just agents; it demands dedicated layers for Observability (monitoring system state), Evaluation (defining metrics of success), and Looping/Control Flow. The outer loop should be deterministic or design-based, while inner loops can be LLM-driven.
The Importance of Validation and QA 17:30
When using generative models for code, the process must include mandatory steps like regression testing, validation testing (e.g., ensuring variables are in config files), and adversarial review (using one agent to critique another's output) to ensure stability.
Addressing Cost and Complexity 53:35
High token burn rates are a major concern. Strategies include using cheaper open-source models, focusing on the initial planning phase (which is costly but simplifies later steps), and implementing independent verifiers to prevent agent chaos.

Watch on YouTube Full article

Jul 26, 2026 · 17:31

The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

The presentation details how poolside scaled their LLM training from smaller models (Laguna M/XS) to a massive 118 billion parameter model for agentic coding. The core strategy involves creating sophisticated synthetic data pipelines—which complement organic data by exposing implicit rationale and structure—and implementing extreme rigor in the pre-training process. Key technical challenges addressed include catching silent failures like broken GPUs, numerical precision loss during tensor parallel accumulation (BF16 to FP32 fixes), and race conditions via weight hashing checks. The resulting model, Laguna S, demonstrates competitive performance against industry leaders on coding benchmarks.

Key takeaways

Synthetic Data for Feature Extraction 10:37
Synthetic data is used not to replace organic data but to complement it by extracting and projecting implicitly hidden features (e.g., implicit rationale, planning, structure). Pipelines are modular, allowing complex tasks to be broken down into stages like multi-stage workflows, cross-domain porting (e.g., math problems to code), or multi-turn chats.
Rigorous Training Validation: Trust Nothing 15:03
To ensure correctness at scale, the team implemented model replica hashing checks; if hashes over weights do not match across distributed replicas, training is immediately halted. Failures observed include silent data corruption from broken GPUs and gradient corruption due to race conditions.
Scaling Success: The 118B Parameter Model
The final model, Laguna S (118B parameters), was trained on 30 trillion tokens across 4,000 GPUs. Early results show strong performance in agentic coding benchmarks (e.g., SWEETbench), outperforming models like GLM 4.5 Air.

Watch on YouTube Full article

Jul 24, 2026 · 21:39

Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

Uber detailed its approach to building closed-loop evaluation systems for a multimodal agent designed to enhance food photography on Uber Eats. The system addresses the challenge of improving image quality at massive scale (90 billion run rate/year) while preserving merchant authenticity and avoiding homogenization. The architecture involves an end-to-end pipeline incorporating Image Understanding/Routing agents, iterative Image Editing agents, multi-dimensional QA gates, and sophisticated continuous learning loops that detect drift and auto-tune the system without human intervention.

Key takeaways

Closed-Loop System Design 17:07
The system operates on a closed feedback loop: production data is sampled, compared against human labels (golden source of truth), mismatches are diagnosed by an 'Umbrella Diagnosis Agent,' and the auto-tuning pipeline adjusts agent configurations. This process ensures continuous improvement without requiring constant human intervention.
Handling Model Drift 20:03
To prevent static models from failing in a live environment, the system uses production data sampling to detect drift. The 'Diagnoser' agent localizes the issue and triggers an auto-tuning pipeline that benchmarks against the golden dataset before deployment.
Evaluation Metrics for Agents 21:05
For routing agents, evaluation involves measuring precision and recall using a confusion matrix (potentially an N x N matrix if multiple branching paths exist). For the enhancement process, 'Pass at K' measures the pass rate after K iterations of refinement.

Watch on YouTube Full article

Jul 23, 2026 · 1:12:03

Coding Agents Are Secretly General Agents

The discussion posits that coding agents are inherently generalist, meaning proficiency in code translates into superior performance across all knowledge work tasks due to a concept called 'positive transfer.' The future of knowledge work is converging on single, integrated platforms (Systems of Record) that provide comprehensive context and surfaces for agent interaction. Key technical advancements include using verifiable code execution environments (like unit testing/linting) as the perfect training ground for agents, leading to autonomous workflows like ticket-to-pull request cycles.

Key takeaways

Coding Agents are Generalist Agents 22:00
The core thesis is that improving an agent's ability to write and execute code makes it better at everything else. This 'positive transfer' capability means agents with coding skills are effectively AGI-complete, as they can write their own tools and interact with various systems.
Verifiability is Key for Agent Training 17:15
Code provides an ideal training ground because its output (e.g., a function, schema) can be programmatically verified (linted or passed through unit tests). This verifiable feedback loop allows agents to learn and refine their performance iteratively, which is crucial for autonomous workflows.
Convergence of Platforms Wins 23:40
The most successful platforms will be those that achieve convergence—integrating context, surfaces, and unit economics into a single system (a 'System of Record'). Fragmentation (e.g., Slack's data walls) is identified as the primary enemy to agentic workflow adoption.
The Future is Autonomous Knowledge Work 26:40
The trend suggests that much of today's office work will be handled by agents. This shift means platforms must evolve from being communication hubs (like Slack) to becoming the central operational layer where all data and tasks reside.

Watch on YouTube Full article

Jun 22, 2026 · 46:40

Logs Are All You Need: Rethinking Observability with AI Agents

Sherwood Callaway introduces Sazabi, an AI-native observability platform designed to disrupt traditional tools like Datadog by focusing on logs as the primary source of truth. The core philosophy is that in the age of coding agents, engineers should interact with production data via natural language chat rather than complex dashboards. Key technical innovations include using Git for persistent agent memory across multiple threads and implementing a read-only sandbox environment to safely execute investigative tasks.

Key takeaways

Logs are Sufficient: Rethinking Observability 22:20
The traditional 'three pillars' (metrics, logs, traces) are considered overkill for modern agentic workflows. By focusing solely on logs, instrumentation becomes significantly simpler, requiring only basic logging statements, as the platform can reconstruct metrics and traces from log data.
AI Agents Generate Alerts, They Don't Evaluate Them 28:00
Instead of using AI to triage noisy alerts (alert fatigue), Sazabi autonomously generates actionable alerts directly from logs and codebase analysis. The agent determines what is meaningful to the user at runtime.
Agent Memory via Git for Shared State 34:05
Sazabi maintains persistent, shared memory across multiple parallel sub-agents and threads by committing findings (e.g., issue lists, facts) to a dedicated Git branch within the sandbox environment. This allows agents to benefit from collective findings.
Sandbox Isolation and Read-Only Access 30:30
The platform operates in a read-only system with no public internet access, ensuring security. All actions are routed through an isolated sandbox environment (e.g., using `bash` tools) to prevent data exfiltration or unauthorized changes.

Watch on YouTube Full article

Jun 19, 2026 · 42:20

Autonomous Agents at Work: From OpenClaw Hype to Enterprise Reality

Autonomous agents represent a significant shift from simple chat interfaces to systems that actively perform actions. To transition these agents from experimental hype (like the OpenClaw movement) to reliable enterprise production models, organizations must implement rigorous governance and control frameworks. PwC outlines a comprehensive approach focusing on risk classification, establishing a minimum control stack (Identity, Input/Output Controls, Auditability), and implementing multi-faceted evaluation processes across Quality, Performance, Safety, Cost, and Business Impact.

Key takeaways

3-Tier Work Classification for Risk Management 1:45
Agents must be classified based on the potential blast radius: 1) Reversible work (e.g., ticket enrichment); 2) Sensitive work (affecting system stability, requiring tighter controls); and 3) Consequential work (touching legal or customer policy documents, highest risk).
The Minimum Control Stack for Production Agents 4:00
Before deployment, four non-negotiable controls must be in place: Agent Identity (credentials treated as first-class data with strict expiration/authorization); Input Controls (guardrails against prompt injection and ensuring tool allow-listing); Output Controls (limiting tool calls, retries, and preventing toxic output); and Auditability.
Five Pillars of Agent Auditability 5:10
Auditing must go beyond simple logging. A comprehensive framework requires monitoring Quality (using LLM-as-judge), Performance (focusing on P99 latency), Safety (PII redaction/filters), Cost (tracking expenditure at the run level), and Business Impact (logging the agent's chain of thought decision process).
Ownership and Architecture are Paramount 8:00
Engineers must maintain ownership over the system architecture, even if AI generates the code. The core logic and blueprints must be human-owned to ensure accountability and proper review processes.

Watch on YouTube Full article