The Video Signal

Topic

LLM Agents

All digests tagged LLM Agents

What is LangSmith? thumbnail

· 5:33

What is LangSmith?

LangSmith is a comprehensive platform designed for the Agent Development Lifecycle (ADLC), enabling build engineers to build, test, deploy, and monitor LLM applications and agents. It functions as a tracing backend, providing crucial observability into complex agent behavior—which can involve dozens of model and tool calls—by tracking every step, diagnosing bugs, and facilitating continuous quality assurance through structured testing and production monitoring.

Key takeaways

  1. Agent Observability is Critical

    Agents are inherently 'black boxes'; LangSmith solves this by providing visibility into the sequence of model calls and tool decisions, which are not visible in the final output.

  2. Tracing Components 0:01

    LangSmith defines three components: a 'Run' (a single unit of work, e.g., one model call or tool call), a 'Trace' (a full pass through the agent, composed of multiple runs), and a 'Thread' (a conversation grouping multiple traces from one customer interaction).

  3. Testing and Validation Loop 0:02

    The platform uses Datasets (sets of examples), Evaluators (which score examples, potentially using an LLM-as-a-judge), and Experiments (running agents over datasets) to verify fixes and compare performance changes (regression testing).

  4. Production Monitoring 0:03

    In production, LangSmith allows online evaluators to score live traffic, generating dashboards that track scores, volume, latency, errors, and cost, and can trigger alerts or route traces to annotation queues.

Watch on YouTube Full article

The Dark Arts of Skill Engineering — Paul Bakaus, Renaissance Geek thumbnail

· 1:04:53

The Dark Arts of Skill Engineering — Paul Bakaus, Renaissance Geek

The talk, 'The Dark Arts of Skill Engineering,' argues that effective AI skill development moves beyond simple prompt engineering and into 'harness engineering.' The speaker details advanced techniques—such as using blind sub-agents, implementing anti-attractors, and utilizing mixture-of-experts architectures—to build robust, deterministic, and highly controllable skills. These techniques focus on exploiting the underlying capabilities of the AI model's execution environment (the 'harness') rather than just the input prompt, resulting in skills that are less susceptible to model drift and more reliable across different deployment environments.

Key takeaways

  1. Harness Engineering vs. Prompting 0:06

    Skills should be treated as extensions of the entire coding/AI harness, not just packaged prompts. This involves leveraging system capabilities like hooks, scripts, and standard output to enforce behavior, which is more reliable than prose instructions. (6:25)

  2. Adversarial Sub-Agents 0:08

    To achieve balanced critique or review, use two or more sub-agents that are blind to each other's output. This prevents a single model thread from anchoring on its own work or failing to detect obvious flaws. (7:55, 10:11)

  3. Anti-Attractors for Divergence 0:17

    To prevent model convergence on common or predictable outputs (the 'median'), implement anti-attractors. Techniques include shaving the next predicted token, generating random seeds (e.g., using celebrities for shaders), or running scripts that introduce unexpected input. (16:59)

  4. Passive Guardrails via Hooks 0:20

    Use hooks (e.g., pre-tool use hooks) to enforce design or code standards (like design linting) by actively preventing the model from writing non-compliant code, rather than relying on post-hoc corrections. (20:38)

  5. Compounding Context and Memory 0:30

    Skills can be made aware of prior sessions by saving intermediate results (e.g., critique files) in a dedicated folder. This allows subsequent runs to build context and track progress over time, enabling multi-session refactoring. (30:00)

Watch on YouTube Full article

Middleware for Managed Deep Agents thumbnail

· 4:42

Middleware for Managed Deep Agents

Middleware is a mechanism for extending the lifecycle of Managed Deep Agents, allowing developers to implement custom behaviors such as policy enforcement, fault tolerance, and rate limiting. The demonstration covers two primary use cases: using prebuilt middleware (like `PIIMiddleware`) to redact sensitive data before it reaches the LLM, and building custom middleware from scratch using decorators like `wrap_tool_call` for logging and auditing tool usage.

Key takeaways

  1. Middleware Functionality

    Middleware extends the agent's lifecycle to manage behaviors like policy enforcement, fault tolerance, and rate limits when interacting with tools or the LLM.

  2. PII Redaction Demo 0:01

    Using prebuilt `PIIMiddleware` automatically detects and redacts sensitive information (e.g., customer emails) from the input, preventing the data from reaching the LLM or being stored in LangSmith.

  3. Custom Middleware Development 0:03

    Custom middleware can be built using decorators (e.g., `wrap_tool_call`) and hooks to intercept and log events, such as every tool call, at specific points in the agent's process.

Watch on YouTube Full article

Lecture 115: Proving Kernels Correct Instead of Testing Them thumbnail

· 40:07

Lecture 115: Proving Kernels Correct Instead of Testing Them

This lecture details the necessity and methodology of formally verifying GPU kernels, moving beyond traditional testing due to the massive input space and non-deterministic nature of GPU execution. The core approach involves reasoning about kernel correctness at the PTX level using Satisfiability Modulo Theories (SMT) solvers. The process requires modeling mathematical concepts (like floating-point operations and unbounded integers) as abstract 'reals' to prove equivalence between a candidate kernel and a reference implementation, thereby guaranteeing algorithmic correctness.

Key takeaways

  1. Limitations of Testing GPU Kernels 1:30

    Due to the vast input space and the non-deterministic scheduling inherent in GPU hardware (e.g., Nvidia), testing alone cannot guarantee correctness. Errors can occur across various boundaries (e.g., denormal boundaries, cancellation terms), and the execution order cannot be guaranteed at runtime. (0:01:30)

  2. Formal Verification at the PTX Level 2:10

    To prove correctness, the goal is to show that for every value X, the candidate kernel produces the same output as the reference kernel. This verification is performed at the PTX level because it provides relatively well-defined semantics, allowing the system to cover inputs from various sources (Triton, CUDA, inline PTX). (0:02:10)

  3. SMT Solvers and Mathematical Abstraction 3:20

    Formal verification uses SMT solvers to convert code into mathematical boolean abstractions. By modeling values as 'reals' (abstract values with infinite precision) and unbounded mathematical integers, the system can prove algorithmic equivalence without requiring bit-exact checks, which would lead to an unmanageable 'explosion' of clauses. (0:03:20)

  4. Advanced Verification Boundaries 4:30

    Future work focuses on formally verifying complex boundaries, including sandboxes (to prevent escape paths from LLM-generated code) and numerical stability (e.g., quantization stability), which are addressed separately from algorithmic correctness. (0:04:30)

Watch on YouTube Full article

Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI thumbnail

· 3:36

Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI

This walkthrough demonstrates a scalable workflow for analyzing high volumes of production traces by leveraging the LangSmith CLI and a coding agent. The process uses a 'perceived error' evaluator to flag potential failures, which are then systematically classified into defined issue types (e.g., flawed plan, context explosion). The final output is a structured, native thread dataset in LangSmith, organized into separate dataset splits for each issue type, making the failures readily available for evaluation metrics or post-training data.

Key takeaways

  1. Automated Failure Identification

    Use the 'perceived error' evaluator to flag threads containing evidence of agent mistakes or misunderstandings, transforming manual review into a data-driven process.

  2. Systematic Failure Classification 2:04

    Define specific agent issue types (e.g., Agent looping, flawed plan) and use a coding agent to classify hundreds of flagged threads into these categories.

  3. Building Structured Datasets

    The workflow culminates in creating a native thread dataset in LangSmith, ensuring a dedicated dataset split for every identified issue type. This provides a sorted, reusable corpus of failures.

Watch on YouTube Full article

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI thumbnail

· 1:03:26

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

This talk provides a deep dive into context engineering for large language model (LLM) agents, focusing on optimizing performance and cost in real-world applications like an AI tutor. The core finding is that compaction techniques (like summarization) are often detrimental because they invalidate the prompt cache, forcing the system to pay full price for every token. For optimal recall and cost efficiency, retaining the full history—especially when leveraging advanced caching mechanisms—is superior to aggressive context reduction.

Key takeaways

  1. Compaction is a potential trap due to Prompt Caching 22:06

    When using prompt caching (which can make cached tokens up to 50 times cheaper), any transformation or summarization of the context invalidates the cache, forcing full-price token usage. Compaction must shrink the context by more than 50 times just to pay for itself.

  2. Full History Retention Wins on Recall 33:33

    Experiments showed that leaving the full history untouched (the 'full history' preset) provided the best memory recall and overall performance, even though it was the most expensive option. Aggressive compaction techniques significantly degraded factual retrieval.

  3. Hybrid Search is Superior for Retrieval 22:48

    For knowledge base browsing, a hybrid search combining semantic similarity (embedding model) with keyword search (BM25) is necessary to achieve high recall. Pure dense retrieval failed when facts were buried at large context sizes (e.g., 400k tokens), while BM25 maintained 100% accuracy.

Watch on YouTube Full article

Building Docs for Agents, Not Humans: Inside OpenWiki thumbnail

· 16:52

Building Docs for Agents, Not Humans: Inside OpenWiki

OpenWiki is an open-source Command Line Interface (CLI) designed to automatically generate and maintain repository documentation specifically optimized for consumption by coding agents. Unlike human-centric wikis, OpenWiki structures content into self-contained, highly searchable snippets using the Open Knowledge Format (OKF). It integrates deeply into a codebase via GitHub Actions, ensuring that documentation remains current with every code change while minimizing manual effort.

Key takeaways

  1. Built for Agents, Not Humans 5:04

    OpenWiki's core thesis is that since agents are increasingly writing code, the documentation must be structured for agent retrieval. This means content must consist of self-contained snippets with predictable headings and optimized context window usage, rather than long narrative pages.

  2. Automatic Maintenance via CI/CD

    The CLI facilitates automatic documentation updates by writing a GitHub Actions workflow. This action runs periodically (e.g., daily), checks the Git history, and uses an agent to generate or update the wiki based on code changes, minimizing manual intervention.

  3. Adoption of Open Knowledge Format (OKF) 11:48

    The system adopts OKF (Google's Open Knowledge Format) by adding a deterministic YAML front matter to every markdown file. This includes fields like `type`, `title`, and `description`, which significantly improves retrieval, filtering, and searching capabilities for agents.

  4. Performance Gains in Benchmarks 15:00

    Early evaluations using the DeepSWE benchmark show that OpenWiki usage leads to fewer tool calls, fewer searches, and a significant drop in token consumption for coding agents while maintaining or improving results.

Watch on YouTube Full article

Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs thumbnail

· 18:08

Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs

The talk analyzes the necessity of undertaking a major codebase refactor—specifically collapsing over ten legacy repositories into a monorepo—despite rapid advancements in AI coding agents. The speaker argues that while modern LLMs (like Opus 4.8) significantly accelerate tasks compared to older models (o3), the business value gained from proactively addressing technical debt and establishing robust development patterns outweighs waiting for perfect AI tooling. Key findings include benchmarking model performance on refactoring tasks, noting that Sonnet 4.6 required one iteration while Opus 4.8 achieved near single-pass completion. The speaker also cautions against relying solely on LLMs to complete complex tasks without proper guardrails and validation.

Key takeaways

  1. Refactoring is necessary despite AI progress

    The refactor was worthwhile because it improved business metrics (commit velocity, time-to-market) by establishing clean patterns, even if models continue to improve.

  2. LLM performance benchmarks show rapid improvement 9:30

    Refactoring tasks that took three hours of back and forth with o3 could be accomplished in roughly one-fifth the time using modern models like Sonnet 4.6 (one iteration) or Opus 4.8 (near single pass).

  3. Monorepos simplify development flow

    Consolidating ten separate repositories into a monorepo allowed almost every developer to contribute, even outside their area of expertise, significantly boosting commit velocity and collaboration.

Watch on YouTube Full article

The Agent Development Lifecycle 101 by Harrison Chase thumbnail

· 45:17

The Agent Development Lifecycle 101 by Harrison Chase

The Agent Development Lifecycle outlines a systematic approach for moving AI agents from isolated demos to reliable production systems. The process is broken down into five stages: Build, Test, Deploy, Monitor, and Govern. Key focus areas include ensuring agent reliability at scale by implementing durable execution, managing complex state via virtual file systems, and using advanced observability tools like tracing and online evaluation (evals) to detect failures and drive continuous improvement.

Key takeaways

  1. Systematic Iteration is Key 3:50

    Successful teams treat agents not as one-off projects but as systems requiring systematic iteration across the entire lifecycle: build, test, deploy, monitor, and improve. The primary challenge in shipping agents reliably at scale is ensuring consistent behavior.

  2. Agent Development Components 5:50

    The core components are Build (frameworks/harnesses), Test (data sets, metrics, benchmarks like Terminal Bench 2), Deploy (durable execution, sandboxes), Monitor (tracing, online evals), and Govern (cost control, tool access management).

  3. The Role of Tracing and Observability 17:06

    Tracing is fundamental for debugging agents, allowing developers to see the inputs and outputs at every step (including tool calls) to understand why an LLM or agent failed. Online evals extend this by scoring production traces without needing ground truth.

  4. Self-Improving Agents 31:30

    Advanced platforms, like LangSmith Engine, are beginning to automate the improvement loop. They run in the background over existing traces, clustering issues and suggesting fixes (code or prompt changes), thereby drastically lowering the burden of operating agents at scale.

Watch on YouTube Full article

Trace Every Cursor Agent Turn in LangSmith thumbnail

· 3:04

Trace Every Cursor Agent Turn in LangSmith

This walkthrough details how to integrate LangSmith tracing with Cursor agents, ensuring that every agent turn is captured as a full, inspectable trace in LangSmith. The process involves installing the LangSmith plugin in Cursor, setting three specific environment variables (including `LANGCHAIN_TRACING_V2`), and running an end-to-end task to verify the hook's functionality.

Key takeaways

  1. Full Trace Capture

    By implementing this setup, every agent turn executed by Cursor is logged as a distinct trace in LangSmith, providing comprehensive visibility into the agent's execution flow.

  2. Trace Structure Details 1:43

    Each trace captures the model run details (model name, token usage), input/output, tool runs (e.g., file reads, shell commands), and nested tasks if sub-agents are involved.

  3. Cross-Agent Comparison

    LangSmith maintains a common trace structure that allows users to compare traces from multiple agents (Cursor, Claude Code, and Codex) within the same workspace.

Watch on YouTube Full article