The Video Signal

Topic

Agent Development

All digests tagged Agent Development

Building a Harness with Jev thumbnail

· 9:15

Building a Harness with Jev

This talk introduces Jev, a new System 1 model from TypeSafe AI, designed for fast, structured decision-making. Unlike traditional LLMs (System 2), Jev does not generate text but instead evaluates a state and questions to return typed answers and probabilities, making it significantly faster (up to 200x) and cheaper (up to 400x) for classification-style tasks. Jev can be integrated into agent harnesses via LangChain's TypeSafe integration to enhance model routing, implement auto-mode for risk assessment, and function as a highly efficient judge for online evaluations.

Key takeaways

  1. Jev as a System 1 Model 2:18

    Jev is a System 1 model that evaluates a state and questions to return typed answers and probabilities, rather than generating text. This makes it ideal for specialized, structured decision tasks.

  2. Performance Advantage 2:45

    Jev is claimed to be 20 to 200 times faster and 40 to 400 times cheaper than LLMs for classification-style tasks.

  3. Three Question Types 6:30

    Jev can answer three types of questions: Choice (multiple choice), Score (on a scale), and Boolean (yes/no). It can process multiple questions from a single state in parallel.

  4. Use Case: Model Routing 8:00

    Jev can assess a given prompt against criteria to help decide whether a fast/cheap model or a more powerful/expensive model should be used, optimizing agent performance.

  5. Use Case: Jev as a Judge

    Jev can score an agent's answer against a provided rubric (e.g., correctness, grounding) for online evaluations, offering a cheaper, faster, and more consistent alternative to LLM-as-a-judge methods.

Watch on YouTube Full article

Accelerate the self-improving AI loop with CoreWeave ARIA thumbnail

· 8:44

Accelerate the self-improving AI loop with CoreWeave ARIA

CoreWeave ARIA is an AI research and iteration agent integrated into Weights & Biases (W&B) designed to accelerate the self-improving AI loop. It addresses common challenges in AI development, such as stalled iteration cycles, massive data volume analysis, and manual dashboard creation. ARIA automates auto-research, analyzes training metrics and agent traces, generates comprehensive reports with suggested next steps, and assists in optimizing LLM prompts and agent performance.

Key takeaways

  1. Automated Auto-Research Loop

    ARIA can conduct auto-research by analyzing recorded training metrics and agent traces to uncover hidden insights. It generates visualization-packed W&B reports and automatically launches follow-up training experiments based on its findings, minimizing manual effort (5:51).

  2. Agent Performance Optimization

    ARIA supports agent development by analyzing production traces and suggesting improvements. It can specifically help refine system prompts and evaluate multiple prompt alternatives using defined datasets to achieve higher quality results at lower latency (7:07).

  3. Comprehensive Workflow Support 2:30

    Beyond research, ARIA handles time-consuming manual tasks like providing advice, generating code, and executing commands, all while supporting concurrent conversations that can continue running in the cloud (2:21).

Watch on YouTube Full article

LangSmith Preview Builds: Test agent changes before production thumbnail

· 8:01

LangSmith Preview Builds: Test agent changes before production

LangSmith Preview Builds extend standard software development workflows to AI agents. By automatically creating a temporary deployment for every GitHub Pull Request (PR), teams can test agent changes and validate functionality in a production-like environment without impacting the main branch. This system ensures that agents are thoroughly tested on preview branches before being merged, while also guaranteeing automatic teardown of the temporary deployment upon PR merge.

Key takeaways

  1. Preview Builds for Agents

    LangSmith automatically generates a temporary deployment for every pull request, allowing teams to test agent changes safely before they reach production. This brings standard software development workflows (PR testing) to AI agents.

  2. Local Testing and Validation

    Agents can be tested locally using `langgraph dev`, which automatically opens LangSmith Studio for validation. Changes made in the local project are reflected in the preview deployment.

  3. Automated Revisioning

    The system supports continuous iteration: every new commit added to a PR triggers a new revision of the preview deployment, allowing for granular testing before final merge.

  4. Automatic Cleanup (DevEx)

    Once the pull request is successfully merged into the main branch, LangSmith automatically tears down and removes the temporary preview deployment, ensuring clean resource management.

Watch on YouTube Full article

Managed Deep Agents - Tools thumbnail

· 6:21

Managed Deep Agents - Tools

This video details how to extend the functionality of a managed deep agent by implementing custom tools. Tools allow agents to interact with external systems (like databases or proprietary APIs) beyond built-in capabilities. Custom tools are defined as standard Python/TypeScript functions decorated with `@tool` and require detailed docstrings, which guide the Large Language Model (LLM) on how and when to use them.

Key takeaways

  1. Purpose of Tools

    Tools give agents capability by allowing interaction with the outside world, such as looking up data in databases or taking actions via external APIs. Built-in tools (e.g., web search) are provided by the underlying model, while custom tools address specific organizational needs.

  2. Defining Custom Tools

    In Python, a custom tool is defined as a function decorated with `@tool` from `LangChain tools`. The function's name becomes the tool name, its parameters are what the LLM must fill out, and the docstring serves as the primary description for the agent.

  3. Integration Process 2:00

    To use a custom tool, define it in a separate file (e.g., `tools/lookup.py`), and then import and pass the function reference into the agent definition script.

Watch on YouTube Full article

Managed Deep Agents - Skills thumbnail

· 8:29

Managed Deep Agents - Skills

Skills allow managed deep agents to access specialized contexts and executable scripts dynamically, moving beyond static instructions. Skills are defined in a structured folder (`skills`) using `skill.md` files, which utilize 'progressive disclosure'—only basic information is given to the LLM initially. This system enables agents to perform complex tasks, such as asking follow-up questions or running Python scripts, and allows for centralized management of these skills via Context Hub without requiring code redeployment.

Key takeaways

  1. Skills vs. Instructions

    While instructions are always visible in the agent context, skills provide specialized, optional contexts that the agent can read or execute only when needed (progressive disclosure).

  2. Skill Definition Structure 2:05

    Each skill resides in a dedicated folder under `skills/` and requires a `skill.md` file. This file must contain front matter fields: `name` (the skill name) and `description` (what is shown to the LLM).

  3. Executing Scripts 5:40

    Skills can contain executable scripts (e.g., Python files like `hello.py`). The agent can identify, read, execute these scripts, and use their output as part of its response.

Watch on YouTube Full article

How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads thumbnail

· 19:29

How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads

Building reliable AI agents requires a comprehensive evaluation loop that extends far beyond prompt engineering. The process involves optimizing foundational tools, establishing rigorous evaluation workflows (Evals), and iteratively refining behavior using feedback signals rather than simple scorecards. Key strategies include starting with small-scale, intuition-based evaluations to identify failure patterns early, analyzing agent trace logs for reasoning failures, and focusing on systemic patterns of error rather than isolated instances.

Key takeaways

  1. Agent Reliability Loop

    Agent behavior is a function of its capabilities, guardrails, and the evaluation process. The system relies on an iterative loop: Prompts $\rightarrow$ Evals $\rightarrow$ Iteration $\rightarrow$ Feedback.

  2. Foundational Optimization

    Before large-scale agent evals, optimize the core set of LLM-friendly tools. Consider implementing an independent critique agent with a remediation loop to fill gaps in base tool limitations.

  3. Starting Small (Vibing) 3:35

    Instead of immediately building comprehensive, scalable evals, start with an intuition-based approach ('vibing'). This allows for quick iteration and radical architectural changes without being hindered by strict scalability requirements.

  4. Deep Failure Analysis 7:50

    To understand failures, analyze agent trace logs to see the reasoning behind a conclusion (e.g., why an agent removed a disclaimer despite explicit instructions). This is more valuable than simple pass/fail metrics.

  5. Scaling Evals and Consensus 10:10

    When involving human raters (scale raiders), provide clear rubrics, detailed examples, and train the team on expected outcomes. For complex systems, collect explanations from raters rather than just a binary pass/fail rating.

  6. Focusing on Patterns 15:30

    When diagnosing model performance degradation (regressions), focus on identifying systemic patterns of failure across the entire golden set, rather than hyper-fixingating on single, isolated examples.

Watch on YouTube Full article

Agent Context Carryover in Livekit Tutorial thumbnail

· 8:07

Agent Context Carryover in Livekit Tutorial

This tutorial demonstrates 'Agent Context Carryover,' a feature released for LiveKit using AssemblyAI's Universal 3.5 Pro model. This capability automatically feeds conversational context into the speech-to-text model, significantly boosting transcription accuracy—especially for proper nouns and key terms—without requiring manual plumbing or explicit context prompting from the developer.

Key takeaways

  1. Agent Context Carryover Functionality

    The feature automatically provides conversational context to the model, improving transcription accuracy when building voice agents on LiveKit. This is achieved by enabling a single parameter rather than implementing complex key term handling.

  2. Implementation Simplicity

    AssemblyAI's LiveKit plugin handles the necessary context plumbing automatically, allowing developers to gain accuracy benefits without setting up custom logic for key terms or sending agent messages manually.

Watch on YouTube Full article