The Video Signal

Topic

LangSmith

All digests tagged LangSmith

How to go from your agent's traces to a fine-tuned model in one workflow thumbnail

· 9:01

How to go from your agent's traces to a fine-tuned model in one workflow

This walkthrough introduces LangSmith Fine-Tuning and the SmithTune CLI, providing a single, end-to-end workflow to convert agent traces (trajectories) into custom, fine-tuned models. The process involves dataset creation and preparation from LangSmith trajectories, training using providers like Fireworks or Baseten, and subsequent evaluation, all managed through the SmithTune CLI. The overall workflow is: Create $\rightarrow$ Prepare $\rightarrow$ Plan $\rightarrow$ Train $\rightarrow$ Evaluate $\rightarrow$ Deploy.

Key takeaways

  1. Supervised Fine-Tuning (SFT) Purpose

    SFT teaches a model desired behavior (e.g., tool selection, workflow adherence) using 'golden examples' derived from agent traces, improving task accuracy and tool reliability.

  2. Trajectory Format Importance 0:01

    The LangSmith trajectory format is critical because it provides a detailed reconstruction of the agent's context, including tools available at each turn, which is necessary for post-training jobs.

  3. SmithTune Workflow Overview 0:02

    The process starts by pulling trajectories from a LangSmith tracing project, defining a task-specific rubric, having a model council judge the traces, and then pushing the approved traces back to LangSmith to form a training dataset.

  4. Training and Evaluation 0:07

    After preparing the dataset (specifying provider, base model, and splits), the user runs `smithtune plan` to review settings, followed by `smithtune train` to execute the job. Evaluation compares the fine-tuned model against the base model on held-out test data.

  5. Deployment 0:08

    If results meet quality targets, `smithtune deploy` is used to serve the tuned model and connect it to the application. Otherwise, the process iterates by refining examples or training settings.

Watch on YouTube Full article

Schedules for Managed Deep Agents: Cron jobs, prompts, and Slack delivery thumbnail

· 3:26

Schedules for Managed Deep Agents: Cron jobs, prompts, and Slack delivery

This video demonstrates how to implement automated, recurring tasks using Schedules for Managed Deep Agents. By configuring cron jobs, agents can run autonomously to generate and deliver structured reports (e.g., a weekly Salesforce pipeline summary) to external channels like Slack, without manual intervention. The process involves defining the schedule syntax, deploying the agent to LangSmith, and ensuring the agent has access to necessary tools and context.

Key takeaways

  1. Automated Reporting via Cron Jobs

    Schedules allow agents to send recurring messages that invoke tools or leverage context, such as generating a weekly pipeline summary for the sales team.

  2. Schedule Configuration 0:01

    Schedules are defined by creating a file in the schedules directory, specifying a cron job, a time zone, a prompt, and optional input context.

  3. Deployment and Monitoring 0:01

    The scheduled agent must be deployed to LangSmith using a deploy command, allowing users to inspect the configured cron jobs and prompts within the LangSmith UI.

Watch on YouTube Full article

How To Build A Harness With Jev | A LangChain x TypeSafe Conversation thumbnail

· 48:04

How To Build A Harness With Jev | A LangChain x TypeSafe Conversation

This conversation introduces Jev, a novel System 1 model from TypeSafe AI, designed for fast, low-latency, and cost-effective decision-making. Unlike traditional LLMs, Jev does not generate text; instead, it makes structured decisions based on well-scoped questions and type-safe schemas. For build engineers, the primary focus is integrating Jev into agent harnesses (using frameworks like LangChain) to enhance reliability, manage risk (e.g., safety classification), and optimize resource usage (e.g., model routing) within the agent development lifecycle.

Key takeaways

  1. Jev is a System 1 Model, not a Generative LLM 5:23

    Jev is fundamentally different from LLMs; it does not generate text or images. It is designed to make fast, intuitive decisions from specific, well-scoped questions using type-safe schemas. This aligns with System 1 thinking (quick determinations), contrasting with System 2 thinking (multi-step reasoning, often associated with LLMs and code).

  2. Three Core Question Types for Structured Decisions 20:13

    Jev supports three decision types: **Choice** (selecting one option from a list, analogous to a classifier); **Score** (rating a state on a single, semantically defined axis, e.g., 0 to 2 for frustration); and **Noul** (a simple True/False or binary check). Questions must be broken down to the point they cannot be broken down further.

  3. Integrating Jev into Agent Middleware 32:37

    Jev can be integrated into agent harnesses as middleware to solve critical problems: **Safety Classification** (e.g., determining if a tool call is destructive and requires explicit intent) and **Model Routing** (selecting the optimal model—fast vs. in-depth—for a given task). This is significantly cheaper and faster than using heavy LLMs for every decision point.

  4. Context Engineering and Observability Best Practices

    When using Jev, it is crucial to minimize the state input to only what is strictly necessary for the determination. Furthermore, building auditability into the system and utilizing platforms like LangSmith for observability and evaluation (Evals) is highly recommended.

Watch on YouTube Full article

Ship a GitHub PR From a Slack Message with Managed Deep Agents thumbnail

· 3:27

Ship a GitHub PR From a Slack Message with Managed Deep Agents

This walkthrough introduces Patch, an agent built using LangChain's Managed Deep Agents (MDAs). Patch automates the process of converting natural language feature requests from a Slack message into a fully drafted GitHub Pull Request (PR), complete with descriptions and code changes. The agent successfully demonstrated implementing features (e.g., adding a share button, changing the background color) for a Tetris side project, proving that complex integrations—including Slack communication, GitHub interaction, and code sandboxing—can be achieved with minimal code.

Key takeaways

  1. Slack-to-GitHub PR Automation

    Patch allows development discussions held in Slack to immediately trigger the creation of a GitHub PR, eliminating the need for manual PR drafting and code sharing via screenshots.

  2. Multi-System Integration Simplicity 2:09

    Managed Deep Agents simplify connecting disparate systems (Slack, GitHub, Sandbox) into a single agent workflow using only a few lines of code.

  3. Agent Configuration Components 2:31

    The agent's functionality is defined by modular files: `agent.py` (defines the agent name and model, e.g., Claude Sonnet 5), `instructions.md` (specifies goals and procedures), and dedicated connectors for Slack, GitHub, and the sandbox.

Watch on YouTube Full article

Building a Harness with Jev thumbnail

· 9:15

Building a Harness with Jev

This talk introduces Jev, a new System 1 model from TypeSafe AI, designed for fast, structured decision-making. Unlike traditional LLMs (System 2), Jev does not generate text but instead evaluates a state and questions to return typed answers and probabilities, making it significantly faster (up to 200x) and cheaper (up to 400x) for classification-style tasks. Jev can be integrated into agent harnesses via LangChain's TypeSafe integration to enhance model routing, implement auto-mode for risk assessment, and function as a highly efficient judge for online evaluations.

Key takeaways

  1. Jev as a System 1 Model 2:18

    Jev is a System 1 model that evaluates a state and questions to return typed answers and probabilities, rather than generating text. This makes it ideal for specialized, structured decision tasks.

  2. Performance Advantage 2:45

    Jev is claimed to be 20 to 200 times faster and 40 to 400 times cheaper than LLMs for classification-style tasks.

  3. Three Question Types 6:30

    Jev can answer three types of questions: Choice (multiple choice), Score (on a scale), and Boolean (yes/no). It can process multiple questions from a single state in parallel.

  4. Use Case: Model Routing 8:00

    Jev can assess a given prompt against criteria to help decide whether a fast/cheap model or a more powerful/expensive model should be used, optimizing agent performance.

  5. Use Case: Jev as a Judge

    Jev can score an agent's answer against a provided rubric (e.g., correctness, grounding) for online evaluations, offering a cheaper, faster, and more consistent alternative to LLM-as-a-judge methods.

Watch on YouTube Full article

Middleware for Managed Deep Agents thumbnail

· 4:42

Middleware for Managed Deep Agents

Middleware is a mechanism for extending the lifecycle of Managed Deep Agents, allowing developers to implement custom behaviors such as policy enforcement, fault tolerance, and rate limiting. The demonstration covers two primary use cases: using prebuilt middleware (like `PIIMiddleware`) to redact sensitive data before it reaches the LLM, and building custom middleware from scratch using decorators like `wrap_tool_call` for logging and auditing tool usage.

Key takeaways

  1. Middleware Functionality

    Middleware extends the agent's lifecycle to manage behaviors like policy enforcement, fault tolerance, and rate limits when interacting with tools or the LLM.

  2. PII Redaction Demo 0:01

    Using prebuilt `PIIMiddleware` automatically detects and redacts sensitive information (e.g., customer emails) from the input, preventing the data from reaching the LLM or being stored in LangSmith.

  3. Custom Middleware Development 0:03

    Custom middleware can be built using decorators (e.g., `wrap_tool_call`) and hooks to intercept and log events, such as every tool call, at specific points in the agent's process.

Watch on YouTube Full article

How Lyft Increased Its Agent Resolution Rate by 16% with LangSmith and LangGraph thumbnail

· 3:41

How Lyft Increased Its Agent Resolution Rate by 16% with LangSmith and LangGraph

Lyft addressed the challenge of scaling its customer support agent stack by replacing brittle, deterministic agents with a meta-agent architecture built on LangGraph and LangSmith. This new self-serve platform allows non-engineering personnel (PMs and ops) to deploy new agents via simple configuration and prompting, drastically reducing agent build time from six months to one to two weeks. This accelerated iteration cycle resulted in a 16% increase in the customer resolution rate.

Key takeaways

  1. Shift to Self-Service Agent Platform 2:25

    The team created a platform enabling PMs and ops to build and ship agents using domain knowledge and natural language prompting, minimizing the need for code changes (merely a config change).

  2. Architectural Improvement via Meta-Agent 3:35

    The system utilizes a meta-agent where all sub-agents are registered dynamically as nodes in the meta-agent, simplifying the composition and deployment of new agents.

  3. Significant Operational Gains

    The agent build time was reduced from six months to one to two weeks, allowing engineers to focus on complex, foundational improvements while increasing the overall resolution rate by 16%.

Watch on YouTube Full article

Share your Managed Deep Agent with your team using Slack thumbnail

· 5:04

Share your Managed Deep Agent with your team using Slack

This guide details the process of deploying a Managed Deep Agent, initially developed in LangSmith Studio, to a production environment using Slack as the primary interaction layer. The process involves running `slack init` to configure the connection, executing a deployment, completing the Slack authorization step, and finally customizing the agent's appearance and message trigger logic for optimal team integration.

Key takeaways

  1. Agent Deployment Workflow

    To connect an existing agent to Slack, run the `slack init` command, followed by a redeployment. The first deployment requires completing a Slack authorization step to link the agent to the internal workspace.

  2. Monitoring and Tracing 0:01

    When the agent is live, all user interactions (requests) are logged and traceable within the LangSmith dashboard, allowing engineers to monitor complete request traces even though the user only sees the final answer in Slack.

  3. Agent Customization 0:02

    The agent's name, description, icon, and background color can be customized by modifying the setup code in the channels directory and redeploying the agent.

  4. Message Trigger Configuration 0:03

    Two key trigger options exist: manual tagging (default) or 'trigger on all messages' (setting `trigger_on_all_messages` to true). Enabling the latter allows the agent to respond to any message in a dedicated channel, not just those directly addressed to it.

Watch on YouTube Full article

Catch Agent Regressions Before You Ship: Evals for Managed Deep Agents thumbnail

· 8:57

Catch Agent Regressions Before You Ship: Evals for Managed Deep Agents

This session details how to implement robust evaluation (evals) for Managed Deep Agents to prevent performance regressions as the agent's capabilities grow. The process involves using Harbor, which ensures each evaluation runs in a fresh container. Evals are structured into an environment (data/state), a job (instruction), and a check (verifier). The workflow is scaffolded using `mda evals init`, which can be automated by handing the task to a coding agent (e.g., Claude Code). Results and traces are managed and monitored in LangSmith, allowing for continuous evaluation and integration into nightly CI pipelines.

Key takeaways

  1. Purpose of Evals

    Evals serve two primary goals: catching regressions (ensuring changes don't break existing features) and 'hill climbing' (actively improving agent capabilities). The focus is on defining and catching regressions.

  2. Harbor's Role in Evaluation

    Harbor is bundled into Managed Deep Agents and is crucial because it builds an image around the agent, ensuring every eval runs in a fresh container. It also manages test execution to prevent environment pollution.

  3. Anatomy of an Eval 2:15

    An evaluation consists of three parts: the environment (the starting data/state), the job (the instruction, defined in a markdown file), and the check (the verifier that determines if the job was completed adequately).

  4. Automating Eval Scaffolding 2:36

    The command `mda evals init` scaffolds the necessary files (e.g., `task.md`, `Dockerfile`, tests). Using the `-i` flag allows the work to be handed off to a coding agent, which generates the initial eval suite.

  5. Production Monitoring

    Evals can be managed in production by running them nightly within a CI system. Results are logged into LangSmith, allowing developers to continuously track agent performance and stability.

Watch on YouTube Full article

Score Every Production Trace with an LLM Judge, from Your Terminal (LangSmith CLI) thumbnail

· 5:08

Score Every Production Trace with an LLM Judge, from Your Terminal (LangSmith CLI)

This walkthrough demonstrates how to implement automated, large-scale evaluation of AI agents using an LLM-as-a-judge within LangSmith. By setting up an online evaluation, developers can programmatically score incoming chat traces (e.g., checking for user frustration) without manually reviewing thousands of interactions. The process involves using a coding agent to install LangSmith skills and configure the evaluation, followed by managing the sampling rate to control costs.

Key takeaways

  1. Online LLM-as-a-Judge Setup

    Online evaluations allow an LLM to automatically score incoming agent traces based on a defined prompt and rubric (e.g., detecting user frustration). The output is a score and reasoning attached directly to the trace.

  2. Using LangSmith CLI for Automation 0:01

    A coding agent can be instructed to use the LangSmith CLI to create and manage the LLM judge evaluator, automating the setup process.

  3. Cost Management via Sampling Rate 0:04

    The sampling rate can be updated (e.g., from 100% to 50%) to reduce evaluation costs while maintaining continuous monitoring.

Watch on YouTube Full article

Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI thumbnail

· 3:36

Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI

This walkthrough demonstrates a scalable workflow for analyzing high volumes of production traces by leveraging the LangSmith CLI and a coding agent. The process uses a 'perceived error' evaluator to flag potential failures, which are then systematically classified into defined issue types (e.g., flawed plan, context explosion). The final output is a structured, native thread dataset in LangSmith, organized into separate dataset splits for each issue type, making the failures readily available for evaluation metrics or post-training data.

Key takeaways

  1. Automated Failure Identification

    Use the 'perceived error' evaluator to flag threads containing evidence of agent mistakes or misunderstandings, transforming manual review into a data-driven process.

  2. Systematic Failure Classification 2:04

    Define specific agent issue types (e.g., Agent looping, flawed plan) and use a coding agent to classify hundreds of flagged threads into these categories.

  3. Building Structured Datasets

    The workflow culminates in creating a native thread dataset in LangSmith, ensuring a dedicated dataset split for every identified issue type. This provides a sorted, reusable corpus of failures.

Watch on YouTube Full article

Inside Clay's Eval Stack: 300M Agent Runs, One LangSmith Pipeline thumbnail

· 13:01

Inside Clay's Eval Stack: 300M Agent Runs, One LangSmith Pipeline

Clay detailed its agentic stack, featuring Claygent (go-to-market research) and Sculptor (go-to-market engineering workflow orchestration). The discussion emphasized that scaling agents to millions of runs requires a robust evaluation philosophy. Key architectural shifts include adopting a data lake for unified first/third-party data access, implementing comprehensive multi-stage evaluations (from local dev to CI), and leveraging large context models (like Fable) to enable complex, self-iterating agent loops.

Key takeaways

  1. Agentic Systems are Becoming Interfaces

    Clay is evolving into an 'agent interface,' making all parts of the product available via CLI and public API. This creates a flywheel where internal and external agents share the same tools, allowing failures in tool invocation to provide user signal for continuous improvement.

  2. Evals are Non-Negotiable for Scale 3:23

    With billions of runs, comprehensive evaluation suites are critical. Clay's philosophy mandates different levels of testing: low-lift local dev checks, and staging/CI environments that mirror production harnesses.

  3. Data Lake Foundation for Agents

    To handle disparate data sources (LangChain traces, Snowflake, Postgres, ClickHouse), Clay is moving to a data lake architecture. This allows agents to run wild on unified first-party and third-party data while enabling safe shadow builds.

Watch on YouTube Full article

LangSmith Preview Builds: Test agent changes before production thumbnail

· 8:01

LangSmith Preview Builds: Test agent changes before production

LangSmith Preview Builds extend standard software development workflows to AI agents. By automatically creating a temporary deployment for every GitHub Pull Request (PR), teams can test agent changes and validate functionality in a production-like environment without impacting the main branch. This system ensures that agents are thoroughly tested on preview branches before being merged, while also guaranteeing automatic teardown of the temporary deployment upon PR merge.

Key takeaways

  1. Preview Builds for Agents

    LangSmith automatically generates a temporary deployment for every pull request, allowing teams to test agent changes safely before they reach production. This brings standard software development workflows (PR testing) to AI agents.

  2. Local Testing and Validation

    Agents can be tested locally using `langgraph dev`, which automatically opens LangSmith Studio for validation. Changes made in the local project are reflected in the preview deployment.

  3. Automated Revisioning

    The system supports continuous iteration: every new commit added to a PR triggers a new revision of the preview deployment, allowing for granular testing before final merge.

  4. Automatic Cleanup (DevEx)

    Once the pull request is successfully merged into the main branch, LangSmith automatically tears down and removes the temporary preview deployment, ensuring clean resource management.

Watch on YouTube Full article

Managed Deep Agents - Skills thumbnail

· 8:29

Managed Deep Agents - Skills

Skills allow managed deep agents to access specialized contexts and executable scripts dynamically, moving beyond static instructions. Skills are defined in a structured folder (`skills`) using `skill.md` files, which utilize 'progressive disclosure'—only basic information is given to the LLM initially. This system enables agents to perform complex tasks, such as asking follow-up questions or running Python scripts, and allows for centralized management of these skills via Context Hub without requiring code redeployment.

Key takeaways

  1. Skills vs. Instructions

    While instructions are always visible in the agent context, skills provide specialized, optional contexts that the agent can read or execute only when needed (progressive disclosure).

  2. Skill Definition Structure 2:05

    Each skill resides in a dedicated folder under `skills/` and requires a `skill.md` file. This file must contain front matter fields: `name` (the skill name) and `description` (what is shown to the LLM).

  3. Executing Scripts 5:40

    Skills can contain executable scripts (e.g., Python files like `hello.py`). The agent can identify, read, execute these scripts, and use their output as part of its response.

Watch on YouTube Full article

Introducing: LangSmith Tuned Evaluators thumbnail

· 4:11

Introducing: LangSmith Tuned Evaluators

LangSmith Tuned Evaluators provide an automated, cost-effective way to attach quality feedback (signals) directly to production traces and threads for AI agents. These out-of-the-box evaluators analyze agent interactions—such as identifying perceived errors or misunderstandings—and surface failure modes that traditional system error logging misses. LangChain manages the entire evaluation pipeline, including prompt writing, judge model management, and inference infrastructure, allowing teams to focus on agent improvement workflows.

Key takeaways

  1. Automated Quality Feedback

    Tuned Evaluators automatically attach useful feedback signals to production traces and threads, helping identify agent behavior that needs attention (e.g., misunderstood user intent or contradictory answers).

  2. Perceived Error Detection

    The initial evaluator, Perceived Error, analyzes multi-turn conversations to detect potential mistakes by the agent, even when no explicit system error occurs. This signal can be derived from subtle patterns like unresolved outcomes or user pivots.

  3. Turnkey Management

    LangChain handles the entire evaluation lifecycle end-to-end: writing/testing prompts, managing judge models, benchmarking, and running inference infrastructure, eliminating the need for users to manage complex components. (See 0:28)

  4. Implementation Steps 0:12

    To use Tuned Evaluators, an organization admin must first enable the feature in LangSmith settings. After enabling, the evaluator can be attached to specific tracing projects.

Watch on YouTube Full article

Building Governed Agents: A Framework for Cost, Control and Compliance thumbnail

· 52:28

Building Governed Agents: A Framework for Cost, Control and Compliance

As autonomous agents scale in production environments, they introduce significant governance challenges related to cost control, data security, and regulatory compliance. This presentation outlines the necessity of implementing centralized runtime controls—such as an LLM Gateway—across the entire Agent Development Lifecycle (ADLC). Centralized governance is critical for managing unpredictable spend, ensuring reliability through fallbacks and rate limits, and maintaining consistent policy enforcement across diverse agent interactions (LLM calls, tool use, and inter-agent communication).

Key takeaways

  1. The Three Pillars of Agent Governance 5:40

    Production agents introduce three main pressure points: Cost (unpredictable spend from looping/context growth), Reliability (need for fallbacks and clear failure behavior), and Compliance (consistent policy enforcement required by regulated industries).

  2. Centralized Runtime Controls are Essential 8:30

    Governance must be implemented as a central infrastructure layer, rather than being bolted on application-by-application. This centralized approach manages interactions with external services (LLMs, tools) and ensures visibility across the entire system.

  3. Comprehensive Risk Management is Required 12:30

    Governance controls must cover four distinct interaction points: LLM calls (cost/data risk), Tool calls (unintended actions/permissioning), MCP calls (data leaving infrastructure), and Agent-to-Agent calls (identity and compounding errors).

  4. Compliance Requires Multi-Layered Guardrails 23:20

    For regulated industries, governance must address specific regulations like GDPR, CCPA, HIPAA, and the EU AI Act. This involves implementing guardrails for PII/secrets detection, role definition, and audit logging.

Watch on YouTube Full article

Create an agent that can browse the web with Managed Deep Agents and Browserbase's Stagehand thumbnail

· 10:39

Create an agent that can browse the web with Managed Deep Agents and Browserbase's Stagehand

This video demonstrates building a production-ready web browsing agent by integrating Managed Deep Agents (LangSmith) with Stagehand v4 and BrowserStack. The resulting agent can interact with live websites—performing actions like scrolling, clicking, and navigating—using specialized tools exposed by Stagehand to connect the AI model to scalable browser infrastructure.

Key takeaways

  1. Agent Architecture Overview

    The solution uses Managed Deep Agents as the core agent harness (LangSmith), which is augmented with tools from Stagehand v4. Stagehand, in turn, connects to web browsing infrastructure provided by BrowserStack for production-grade scalability.

  2. Stagehand V4 Tools 3:57

    The agent is given three core tools from Stagehand: `screenshot` (visually inspects the rendered page), `snapshot` (inspects the active page and hydrates element IDs for simple interactions), and `run` (accepts snapshot actions or JavaScript via the Playwright-shaped page API, ideal for multi-step workflows).

  3. Development Workflow (Local vs. Production) 6:00

    The development process involves running `mda dev` locally to test the agent in a managed deep agent studio environment. For production, the deployment is finalized using `mda deploy`, which creates a serverless deployment within LangSmith's Context Hub.

Watch on YouTube Full article

How Harmonic 4x'd User Retention by Building on Deep Agents thumbnail

· 16:25

How Harmonic 4x'd User Retention by Building on Deep Agents

Harmonic transitioned its natural language interface, Scout, from a brittle query parsing graph to an architecture built on Deep Agents and a simple model-plus-tools loop. This shift quadrupled week one to week four user retention. The core technical lesson is that robust agent design requires managing context via a 'harness contract,' ensuring that all artifacts (like visualizations or large search result sets) are visible to the model—either in the message list or offloaded through file system tools—to prevent the UX from becoming an invisible black box.

Key takeaways

  1. Deep Agents significantly boost retention 2:04

    Switching to Deep Agents resulted in a fourfold increase in week one to week four user retention for Scout. (1:24)

  2. The agent architecture simplified from graphs to loops 4:01

    Scout evolved from complex, multi-node query parsing graphs (LangGraph) into a simpler model and tools loop, mediated by middleware. (2:41)

  3. Context management is handled by the harness 8:16

    Deep Agents manage context overload using mechanisms like compaction for long message lists and file system abstraction to store large results, returning only pointers to the model. (4:56)

  4. UX must respect the agent's context contract 11:44

    For a product UX to be useful, any rendered element (e.g., charts) must either reside in the message list or be discoverable by the model via tools/file system pointers; otherwise, it is invisible to the agent. (7:04)

Watch on YouTube Full article

Voice Agent observability with LangSmith thumbnail

· 7:38

Voice Agent observability with LangSmith

This session details how to implement robust observability for voice agents built using the Google ADK and Gemini Live model by integrating LangSmith tracing. The process involves defining a custom plugin that captures not only the conversation transcript but also the full audio stream (user input and agent output). This visibility allows engineers to debug complex interactions, analyze tool usage, track interruption events, and monitor token-level costs for production readiness.

Key takeaways

  1. Gemini Live Model Functionality

    Gemini Live is Google's native audio model that operates in a speech-to-speech manner. It takes audio directly as input and produces audio as output without transcribing to text, resulting in low latency and natural, emotive voice quality.

  2. LangSmith for Observability 2:05

    LangSmith is a platform built by LangChain specifically for AI agent observability and evaluations. It provides visibility into the internal workings of the voice agent, which is crucial for safe production deployment.

  3. Capturing Conversation Audio 4:00

    To ensure the 'source of truth' for a voice interaction is captured, the tracing setup must include functions to record both user audio and agent audio. Recording agent audio requires careful placement (e.g., using a `set play callback` on the audio out class) to capture only what the user actually heard.

Watch on YouTube Full article

The Art of Loop Engineering: How to Build Agents That Improve Over Time thumbnail

· 46:08

The Art of Loop Engineering: How to Build Agents That Improve Over Time

The video introduces 'Loop Engineering,' an emerging design pattern critical for building reliable, production-grade AI agents. Agents are inherently non-deterministic; therefore, they require structured loops—such as the Core Agent Loop, Verification Loop, Event-Driven Loop, and Self-Improvement Loop (Hill Climbing)—to ensure reliability, automate continuous improvement, and integrate seamlessly into existing systems. The LangSmith platform is presented as a key tool for managing this complex agent development lifecycle.

Key takeaways

  1. Core Agent Loop (Level 1) 10:30

    This basic action-taking loop involves the model receiving context, calling tools to complete tasks, and receiving observations until completion. Optimization focuses on selecting the right model intelligence for the task complexity and improving tool descriptions via prompt engineering.

  2. Verification/Goal Loop (Level 2) 15:20

    This loop adds reliability by introducing a 'Grader' or verification step. After the core agent attempts a task, the Grader scores the result against predefined criteria (rubrics). If criteria are not met, the process is fed back into the agent loop for correction.

  3. Event-Driven Loop (Level 3) 20:00

    Agents become powerful when triggered by external systems (e.g., Slack messages, emails). This loop integrates the agent into relevant workflows, making it a system improvement mechanism rather than just an isolated task executor.

  4. Self-Improvement Loop / Hill Climbing (Level 4) 23:20

    This advanced loop automates agent improvement by analyzing traces. A helper agent, like LangSmith Engine, detects failure modes (e.g., improper tool arguments, missed context) and autonomously updates the core harness—including prompts, tools, skills, or memory—to improve future performance.

Watch on YouTube Full article