The Video Signal

Topic

Continuous Integration (CI)

All digests tagged Continuous Integration (CI)

· 21:17

Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind

SonderMind details its approach to building Sonder, a clinically grounded AI coach for mental health support. The core focus is on achieving safety and effectiveness through eval-driven development, moving beyond general-purpose LLMs which are unsuitable for high-stakes care. The architecture employs modular guardrails (Input/Output) within a Supervisor/Executor/Evaluator framework. Safety is maintained by integrating clinical judgment into the CI pipeline via annotated traces, ensuring that every model or prompt change is scored against expert-defined 'evals' before deployment.

Key takeaways

  1. Specialized AI for Mental Health 2:00

    General purpose LLMs are insufficient for mental health care due to the high stakes involved. Sonder was designed specifically to act as a front door to human therapy or provide support between sessions, addressing a critical gap in current AI offerings.

  2. Robust Architecture and Guardrails 4:00

    The system uses separate Input and Output guardrails—acting as 'sandwich' layers—to monitor user input and the entire conversation flow for clinical safety risks. This modularity allows iteration without compromising safety.

  3. Clinical Feedback Loop in CI 9:40

    Safety is enforced by capturing edge-case conversations (traces) and having licensed professionals annotate them. These annotations generate typed 'evals' that gate releases, ensuring every model or prompt change is tested against real-world clinical judgment.

  4. Focus on Correct Triggers 6:10

    The guardrails are calibrated not to generate more alerts (false positives), but to correctly identify genuine safety risks. The system must distinguish between a user in an active crisis and one seeking past support.

Watch on YouTube Full article

· 23:13

Evaling Video Slop — Maor Bril, Character.ai

The challenge of evaluating AI-generated video ('video slop') is that traditional metrics like CLIP score are effective for single frames but fail to capture temporal coherence, physical plausibility, or overall narrative quality. The solution involves shifting from absolute scoring (e.g., rating 1-10) to pairwise preference comparison (is B better than A?). This process utilizes Vision Language Models (VLMs), such as Qwen3-VL, trained with Bradley-Terry loss on pairs of real and deliberately broken footage. Critically, this evaluation judge is integrated into the Continuous Integration (CI) pipeline as a regression gate, allowing developers to catch drift and errors much earlier in the generation loop when they are cheapest to fix.

Key takeaways

  1. Limitations of Current Video Metrics

    Tools like CLIP score are excellent for judging individual frames but cannot assess temporal incoherence, story consistency, or physical plausibility across a video's duration. The core problem is that 'video is a storytelling medium,' requiring checks on pacing and narrative flow.

  2. The Shift from Scoring to Comparison 15:20

    Absolute scoring (e.g., rating a video's quality) is unreliable because human judgment varies greatly. The effective approach is pairwise preference, asking which of two videos tells a better story or exhibits fewer artifacts.

  3. Implementing the Evaluation Judge in CI 17:27

    The evaluation judge (e.g., Qwen3-VL) is used as a regression gate within the CI pipeline, ensuring that every AgentX release clears an 'eval wall' calibrated against human scores before reaching users. Catching drift early significantly reduces correction costs.

  4. Training Data Improvement 20:47

    Initial models failed because they scored the 'vibe' (gloss) rather than the substance (story/physics). The fix involved creating a dataset by pairing real, human-generated footage against AI-generated footage to prevent the model from becoming an overfitted 'AI detector.'

Watch on YouTube Full article

· 1:15:02

Full Workshop: Setting Yourself Up for Success —Jason Liu, OpenAI Codex

Jason Liu provides a comprehensive workshop on leveraging Codex and the Agents SDK for advanced knowledge work automation. The focus shifts from simple text generation to using AI agents (threads) as collaborative teammates capable of managing complex projects, interacting with external applications via 'Computer Use,' and maintaining long-running processes through automated 'heartbeats.' Key strategies include building custom skills/plugins, utilizing a personal memory vault, and treating the computer itself as an integrated component of the workflow.

Key takeaways

  1. Systematic Context Management via Appshots 20:05

    The 'Appshot' feature is highlighted as critical for providing deep context to AI models. Unlike simple screenshots, Appshots capture the entire accessibility tree of an application (e.g., Slack), allowing agents to perform complex actions with single function calls and greatly improving accuracy.

  2. Advanced Project Management with Threads 5:50

    The concept of 'compaction' allows threads (pinned conversations) to maintain long-term context, acting like dedicated project managers. These threads can delegate tasks to sub-agents and communicate with each other, forming a collaborative team structure.

  3. Automating Long-Running Workflows (Heartbeats) 11:30

    Instead of manual checks, 'heartbeat' automations can wake up threads over time to perform continuous maintenance tasks (e.g., checking PR status, monitoring support issues), allowing for sustained project oversight.

  4. Extending AI Capabilities with Skills and Plugins 23:25

    Skills are simple constructs (files/scripts) that solve specific problems (e.g., 'Review my code like Charlie'). Plugins connect the system to external services (Slack, Gmail), allowing agents to perform actions across different platforms.

Watch on YouTube Full article

· 46:08

The Art of Loop Engineering: How to Build Agents That Improve Over Time

The video introduces 'Loop Engineering,' an emerging design pattern critical for building reliable, production-grade AI agents. Agents are inherently non-deterministic; therefore, they require structured loops—such as the Core Agent Loop, Verification Loop, Event-Driven Loop, and Self-Improvement Loop (Hill Climbing)—to ensure reliability, automate continuous improvement, and integrate seamlessly into existing systems. The LangSmith platform is presented as a key tool for managing this complex agent development lifecycle.

Key takeaways

  1. Core Agent Loop (Level 1) 10:30

    This basic action-taking loop involves the model receiving context, calling tools to complete tasks, and receiving observations until completion. Optimization focuses on selecting the right model intelligence for the task complexity and improving tool descriptions via prompt engineering.

  2. Verification/Goal Loop (Level 2) 15:20

    This loop adds reliability by introducing a 'Grader' or verification step. After the core agent attempts a task, the Grader scores the result against predefined criteria (rubrics). If criteria are not met, the process is fed back into the agent loop for correction.

  3. Event-Driven Loop (Level 3) 20:00

    Agents become powerful when triggered by external systems (e.g., Slack messages, emails). This loop integrates the agent into relevant workflows, making it a system improvement mechanism rather than just an isolated task executor.

  4. Self-Improvement Loop / Hill Climbing (Level 4) 23:20

    This advanced loop automates agent improvement by analyzing traces. A helper agent, like LangSmith Engine, detects failure modes (e.g., improper tool arguments, missed context) and autonomously updates the core harness—including prompts, tools, skills, or memory—to improve future performance.

Watch on YouTube Full article