The Video Signal

Channel

AI Engineer

Digests from AI Engineer

Trading Desks to Clinical Trials: Parallels in Applied Vertical AI — Ayush Bhardwaj, Allos AI thumbnail

· 20:02

Trading Desks to Clinical Trials: Parallels in Applied Vertical AI — Ayush Bhardwaj, Allos AI

The talk outlines a structured methodology for building and iterating in applied vertical AI, arguing that model infrastructure and APIs are commodities. The true 'moat' lies not in the technology itself, but in proprietary data and deep domain expertise. The process requires narrowing the problem scope, curating unique datasets (like failed experiments or trade theses), and crucially, integrating a human domain expert ('the user') to create an endless learning loop that guides prompt refinement and error analysis.

Key takeaways

  1. The Moat is Domain Expertise 12:30

    In vertical AI, the competitive advantage does not come from the model (e.g., Claude or ChatGPT) or the infrastructure; it comes from proprietary data and domain expertise which must be curated by human experts.

  2. Focus on Narrow Tasks 6:35

    When formulating a problem for an AI agent, do not try to solve everything at once. Break down complex tasks into very narrow, specific steps (e.g., ranking stocks based on IT capital expenditure in US equities).

  3. The Importance of Proprietary Data 8:00

    Generic data sources are insufficient. The most valuable assets are proprietary datasets that are expensive to acquire or withheld by industry players (e.g., failed clinical trials, internal trade theses).

  4. Hire the User, Not Just the Model 15:30

    The single most critical step is hiring a domain expert ('the user'). This person guides the tool by curating sources, refining prompts, and performing judgment—a process that transforms generic AI tools into industry-specific assets.

Watch on YouTube Full article

Healthcare’s Agent Bytecode: X12 as the Harness for AI Agents — Vasant Kearney, Onlay thumbnail

· 20:25

Healthcare’s Agent Bytecode: X12 as the Harness for AI Agents — Vasant Kearney, Onlay

The presentation argues that reliable AI agents in healthcare claims processing must treat X12 not merely as a file format, but as an underlying structural 'harness' or contract. This approach is necessary because various payer systems (phone portals, web interfaces, and X12 feeds) are often built by disparate teams and can contradict each other, meaning no single surface represents the ground truth. By grounding agentic execution in the structured rules of X12—which governs every stage from eligibility (270) to payment (835)—developers can build systems that maintain data integrity until downstream evidence proves otherwise.

Key takeaways

  1. Goal: Cost and Patient Experience 1:46

    The primary objective when solving healthcare problems is twofold: driving overall cost reduction and improving the patient experience. Technical solutions must be grounded in these concepts.

  2. X12 as a Structural Harness 8:16

    Instead of viewing X12 only as a data format, it should be treated as a contract that defines the relationship between providers and payers. This structure guides agentic execution across all claim lifecycle steps (e.g., eligibility check 270 to payment 835).

  3. Enterprise Memory Constraints

    For reliable, large-scale systems in healthcare, memory must be stored in a database rather than on local disk, ensuring logical separation and preventing data loss or contamination.

  4. Skepticism of LLMs

    While AI models are powerful, developers must remain 'AI pilled' yet highly skeptical. Over-reliance on overpowered or expensive models can negate cost savings goals; testing and validation must be rigorous to prevent system failure when introducing new models.

Watch on YouTube Full article

Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia thumbnail

· 19:15

Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia

The talk details how Ufonia built a comprehensive safety and evaluation stack for Dora, a conversational AI used in clinical post-op follow-ups. Because randomized A/B testing is unethical and illegal when dealing with patients, the system cannot rely on reactive rollbacks or standard model benchmarks. Instead, the approach shifts to rigorous simulation (the 'inner loop') using frameworks like Matrix, which employs simulated patients (PatBot) and an expert LLM judge (BevJudge). Safety is proven by optimizing prompts against a cost matrix (e.g., prioritizing sensitivity over overall accuracy) and utilizing automated prompt optimizers like Jeppa, ensuring the system ships evidence, not just a model.

Key takeaways

  1. Safety Constraints in Healthcare AI 3:50

    Standard software safety nets (A/B testing, rollbacks) fail when dealing with patients because randomizing into a worse variant is unethical and illegal; once a call is made, it cannot be undone. The model card's benchmark scores are insufficient defense at post-incident reviews.

  2. The Necessity of Simulation 10:50

    Since real-world testing (the 'outer loop') is too risky, the process must emulate high-reliability industries like self-driving cars. The simulation framework, Matrix, uses an LLM (PatBot) to play the patient against hazards written by clinicians.

  3. Automated Hazard Detection 13:50

    A second LLM, BevJudge, validates simulated dialogues. It is trained and validated against a corpus of 240 examples labeled by 10 clinicians from 10 specialties, achieving expert-level performance (e.g., F1 score of 0.96) with near-perfect sensitivity.

  4. Optimizing Prompts via Cost Matrix 17:00

    Instead of manual prompt engineering, the process uses optimizers like Jeppa (Genetic Pareto), which iteratively updates prompts based on a defined cost matrix. This allows optimization for specific metrics, such as maximizing sensitivity (catching red flags) over general accuracy.

Watch on YouTube Full article

Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health thumbnail

· 21:49

Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health

The talk outlines critical architectural guardrails necessary for deploying member-facing healthcare AI. The core argument is that most safety failures are not model flaws but architectural decisions made before any tokens are generated. Safety must be built into three non-negotiable foundations: protecting PHI at the pipeline boundary, ensuring deterministic code layers handle high-stakes decisions (like emergency routing), and implementing continuous monitoring using multiple signal sources.

Key takeaways

  1. Architectural Failures vs. Model Failures

    Most AI safety failures in healthcare are architectural decisions, not model failures. The system must be designed to prevent failure at the structural level before considering prompt engineering.

  2. Three Non-Negotiable Foundations 3:55

    1) Constraint is the architecture (not just policy). 2) Deterministic rules must belong above the model layer, as anything that can never be wrong cannot be left to probability. 3) Safety must be a continuous evaluation layer, not a one-time gate.

  3. PHI Protection at Ingestion 8:47

    Instead of treating PHI redaction as a runtime problem (on the dashboard), the architecture must strip PHI at the pipeline boundary during ingestion, ensuring it is never stored in the data lake.

  4. Deterministic Code Layer for High Stakes 13:35

    Irreversible decisions (e.g., emergency escalation to 911/988, intent routing) must be handled by a deterministic code layer that runs *before* the LLM processes the turn. The model should not get a vote on high-stakes calls.

  5. Decision Framework: Worst Case Wins

    When stakeholders disagree on a feature launch, severity must be set by the worst plausible outcome (the maximum potential harm), not the average case or current capacity. When unsure, default to the safer mistake.

Watch on YouTube Full article

From Ambient Documentation to Clinical Intelligence — Chaitanya Asawa, Abridge thumbnail

· 21:35

From Ambient Documentation to Clinical Intelligence — Chaitanya Asawa, Abridge

The talk details Abridge's evolution from solving clinical documentation burnout—a high-stakes administrative problem in healthcare—to building comprehensive clinical intelligence tools. The speaker emphasizes that all healthcare processes are downstream of the doctor-patient conversation. Technically, the core challenges involve maintaining extremely high quality and low latency in a high-stakes environment, requiring novel approaches like decomposing complex tasks into smaller models (instead of relying solely on frontier LLMs) and developing sophisticated evaluation systems using expert human judges and rubrics to address the small generator/verifier gap.

Key takeaways

  1. The Centrality of Conversation 5:50

    All administrative processes in healthcare (billing, clinical decision support, etc.) are built around the core conversation between a doctor and a patient. Abridge aims to automate this entire downstream machinery.

  2. The Productivity Paradox in Healthcare 10:20

    Unlike many industries where productivity increases lower costs, administrative costs in healthcare have continued to rise over decades, creating a significant operational burden that technology must address.

  3. High Stakes AI Development

    In clinical decision support, the cost of being wrong is extremely high. This necessitates rigorous quality control and evaluation methods far beyond typical generative AI applications.

Watch on YouTube Full article

Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai thumbnail

· 16:55

Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai

Gabriel Jorge Menezes details the complex infrastructure required to train and serve Krea 2, a diffusion transformer model trained from scratch on thousands of GPUs. The system addresses challenges like silent failures at scale, GPU thermal throttling, and cross-node communication issues by implementing advanced monitoring (tensor core utilization, InfiniBand metrics). For serving, they built a robust architecture using Gang scheduling and Kubernetes features (virtual kubelet, taints/tolerations) to ensure training workloads can utilize the entire cluster while maintaining production uptime through seamless traffic flipping.

Key takeaways

  1. Metrics are essential for large-scale pre-training 9:50

    Do not rely on GPU utilization (which is 'a lie'). Instead, monitor tensor core utilization and collect custom metrics like InfiniBand/NVLink errors, as most failures relate to cross-node communication. [5:58], [6:48]

  2. Embrace failure for stability 4:18

    When scaling training runs, instead of debugging every crash, it is often more efficient to 'let it crash.' The system should be designed to recover and run successfully on the same nodes over extended periods. [4:18]

  3. Checkpointing must be extremely fast 8:29

    To make long training runs survivable, checkpoint aggressively against a high-speed filesystem capable of writing terabytes quickly (e.g., achieving >1 TB/30 seconds). [8:29]

  4. Decouple training and production workloads 11:01

    Use a system that allows high-priority training jobs to utilize the entire cluster while seamlessly migrating inference traffic (production) to external providers or other clusters, ensuring zero downtime. [11:01]

Watch on YouTube Full article

Generative Video at the Speed of Light — Keegan McCallum, uRun thumbnail

· 8:43

Generative Video at the Speed of Light — Keegan McCallum, uRun

The presentation shifts the focus of generative video from improving quality (e.g., Sora) to maximizing efficiency and enabling long-horizon, real-time interaction. The core argument is that the limiting factor is no longer model fidelity but the serving infrastructure required for continuous, synchronized streaming. Key technical requirements include global GPU distribution, WebRTC/ICE/TURN setup, and complex pipelines capable of integrating multiple models (e.g., video-to-video transformation) in real time.

Key takeaways

  1. Shift from Quality to Efficiency 3:55

    The most significant axis of improvement is efficiency and long horizon generation, allowing for continuous streaming at a fraction of the cost compared to slower, high-quality generations. For example, real-time generation can be produced at about 1/100th the cost of minutes-long clips.

  2. Cost and Accessibility 6:24

    Generative video is becoming highly accessible: $10 can now buy roughly three hours of continuous generated video, while $50 could provide up to 15 hours of interaction.

  3. New Interaction Paradigms 7:18

    The technology enables new use cases beyond simple content creation, such as 'magic mirrors' (transforming a webcam feed in real time) and providing visual mediums for users who do not think primarily in text.

Watch on YouTube Full article

Voice agents with Realtime Video — Sidney Primas, LemonSlice thumbnail

· 26:36

Voice agents with Realtime Video — Sidney Primas, LemonSlice

LemonSlice aims to break the Avatar Turing test by creating highly realistic, real-time video avatars. The core technical challenges addressed include achieving emotional expressiveness (requiring specialized audio encoders beyond monotone audiobook data), mitigating error accumulation over extended generation periods (e.g., 8+ hours), and optimizing for real-time performance. A major focus is on the 'model harness'—the complex orchestration of threads and queues across GPU/CPU to ensure uninterrupted, stutter-free video streaming at scale. The company also highlights cost parity, noting that generating high-resolution video costs comparably to running a voice model.

Key takeaways

  1. Real-Time Video Generation Challenges 17:05

    Generating real-time avatars requires training models with an attention mask that enforces looking only into the past, as future frames do not exist yet. Furthermore, speeding up generation involves collapsing many denoising steps (e.g., 30 steps) down to a single step.

  2. Error Accumulation Mitigation 20:22

    A significant technical hurdle is error accumulation, where errors introduced in previous video blocks compound over time. The company claims to have developed a novel method to generate very long videos with no noticeable error buildup.

  3. Model Harness and Cost Parity 23:54

    The most durable value lies in the 'model harness'—the orchestration of threads and queues across GPU and CPU to maintain real-time, stutter-free video. Surprisingly, serving this complex visual layer costs about the same as serving a voice model.

  4. Future Direction: Emotional Engine

    The next generation involves building an 'emotion engine' that predicts and controls emotional reactions and actions based on both audio input and text input, moving beyond current awkward interactions.

Watch on YouTube Full article

While my guitar gently speaks — Todd Fisher, Philo Ventures thumbnail

· 18:35

While my guitar gently speaks — Todd Fisher, Philo Ventures

Todd Fisher details his project to make a guitar speak, showcasing an evolution from simple audio effects to complex AI-driven speech and singing synthesis. The process involves integrating multiple technologies: using JUCE for plugin development, employing advanced signal processing techniques like Energy Gap Segmentation and Sonority Peak Syllabification for word slicing, and utilizing algorithms such as YIN pitch detection and vocoders to achieve synthesized vocals.

Key takeaways

  1. Initial Goal: Making the Guitar Speak

    The project started with the goal of having a guitar respond to questions from the room, moving beyond traditional effects like the talk box. This was initially achieved by playing pre-recorded audio files.

  2. Word Segmentation Challenges 12:30

    Automatically slicing speech into words is challenging because standard methods (like Energy Gap Segmentation, which relies on silence) fail when spoken words run together. The speaker explored Sonority Peak Syllabification as an alternative.

  3. Singing Synthesis Pipeline 17:03

    To make the guitar sing, the process requires detecting the fundamental frequency using the YIN algorithm. This pitch is then used to shape a synthesized note (via ADSR) and passed through a vocoder, which maps it onto pre-baked, pitch-shifted vocal samples.

  4. Advanced Interaction: LLM Integration

    The most advanced step involves using speech-to-text (Whisper) to capture a question, feeding it into a local Large Language Model (LLM), and then playing the resulting text response through the guitar's synthesis system.

Watch on YouTube Full article

Building an Agentic Video Editor for Mass Consumer — Ekaterina Deyneka, Reelful thumbnail

· 12:45

Building an Agentic Video Editor for Mass Consumer — Ekaterina Deyneka, Reelful

Reelful addresses the complexity of video editing by implementing an agentic workflow that automates content creation from raw footage. The system accepts user media and directional prompts, utilizing a multi-stage pipeline—including Media Understanding, Creative Planning, and execution within a sandboxed environment—to generate polished videos. Architecturally, the process is compared to agentic app building, but with the added complexity of judging optimal cuts and structure from messy, real-world footage.

Key takeaways

  1. Agentic Video Editing Definition 2:09

    The user provides raw media and context/directions (e.g., 'add captions,' 'add music'). The agent then autonomously finds usable moments, assembles the clip, and generates supplementary assets like captions, music, voiceovers, and B-roll.

  2. Architectural Parallelism 5:33

    From an infrastructure standpoint, agentic video editing is structurally similar to agentic app building. Both require a prompt/input (media + prompt for video) and execute within a remote 'sandbox' environment by an agent equipped with tools and skills.

  3. The Challenge of Editing vs. Generating 6:50

    Editing real footage is significantly more complex than generating content from scratch, as the agent must actively judge which moments to keep or omit and organize messy, incomplete material into a polished result.

  4. The Full Agentic Pipeline 8:50

    The process involves: (1) Media Understanding/Transcription $ ightarrow$ (2) Creative Plan generation (for user approval) $ ightarrow$ (3) Sandbox execution by the agent (using skills like cut rules, font pairings, B-roll generation) $ ightarrow$ (4) Remotion composition $ ightarrow$ (5) Verification Layer (to ensure renderability).

Watch on YouTube Full article

Training Krea 2: What matters in generative model training — Sangwu Lee, Krea.ai thumbnail

· 21:46

Training Krea 2: What matters in generative model training — Sangwu Lee, Krea.ai

Sangwu Lee details the training methodology for Krea 2, an open-sourced image foundation model, emphasizing that data curation is the most critical factor after architecture is locked. The talk contrasts production models (like ChatGPT-2) which prioritize consistency and reliability but suffer from mode collapse, with Krea 2's focus on stylistic diversity and fast iteration. Key techniques covered include advanced deduplication using hash-based methods (pHash/MD5) followed by embedding-based approaches (SigLip), specialized captioning pipelines to capture contextual details (e.g., a painting being framed), and multi-stage training from low to high resolution.

Key takeaways

  1. Data is Paramount for Model Quality 9:59

    After locking in the architecture, data curation determines model performance. The goal was to maximize stylistic diversity while filtering out undesirable content (e.g., images where captions consistently fail to capture important context) and avoiding reliance on synthetic AI-generated training data due to its 'sticky' aesthetic.

  2. Balancing Consistency vs. Diversity 2:20

    Large production models often achieve high consistency by focusing on the 'average' subject (e.g., a boring average person), leading to mode collapse and limited stylistic range. Krea 2 was designed to optimize for fast generation and broad stylistic exploration, catering to creative studios that are still defining their visual goals.

  3. Advanced Deduplication and Filtering 16:55

    Deduplication runs first on basic hashes (pHash or MD5) across billions of images, followed by embedding-based methods like SigLip to remove near duplicates. Filters are also applied using large vision language models (VLMs), which are then distilled into cheap classifiers for scalable filtering over massive datasets.

  4. Multi-Stage Training Pipeline

    The training process progresses through several stages: low to high resolution pre-training (learning semantics first, then structure/detail), supervised fine-tuning (SFT), preference optimization (collecting user pairs for comparison), and Reinforcement Learning (RL). The final step involves training a prompt expander LLM to generate detailed prompts that align with the model's trained data distribution.

Watch on YouTube Full article

How to Kill the Code Review — Ankit Jain, Aviator thumbnail

· 16:26

How to Kill the Code Review — Ankit Jain, Aviator

The traditional code review process is failing due to increasing code churn and reliance on AI agents, leading reviewers to only skim results. The speaker argues that code review must shift its focus from verifying line-by-line correctness (semantic accuracy) to preserving crucial non-technical aspects like knowledge sharing, mentorship, and architectural alignment. The proposed solution involves capturing the entire development session's decisions (the 'intent'), converting these into formal acceptance criteria, building an AI Slop Registry of recurring feedback, and generating a verifiable test plan that serves as the new review surface.

Key takeaways

  1. The Code Review Crisis 2:07

    Code churn is high (861%), leading to increased bottlenecks at the review stage. Currently, over 30% of changes merge without any review, and even when AI reviews are used, human involvement often reduces to skimming rather than deep reading.

  2. Review Must Preserve Alignment 5:05

    Code review's most critical function is not just catching bugs (semantic accuracy), but facilitating knowledge sharing, mentorship, and architectural alignment. This 'alignment' aspect must survive the shift to automated development.

  3. The New Review Surface: Intent vs. Diff 12:38

    Instead of reviewing code diffs, the review surface should become a verification of 'intent' and 'evidence.' This is achieved by capturing user decisions from development sessions, converting them into acceptance criteria, and generating a test plan that runs against a live preview.

Watch on YouTube Full article

Security Firewall for Agents — Ryan Dahl, Deno thumbnail

· 19:06

Security Firewall for Agents — Ryan Dahl, Deno

The video addresses the critical security challenge of granting autonomous AI agents write access to sensitive production systems (e.g., Postgres, Kubernetes). Since relying solely on model alignment is insufficient against prompt injection, the speaker introduces Claw Patrol: an MIT-licensed proxy that operates below the HTTP layer. This system inspects every byte leaving the agent, enforcing granular rules defined in HCL (Terraform configuration language) to prevent destructive actions like spawning `psql` subprocesses and dropping tables.

Key takeaways

  1. The Danger of Autonomous Agents 5:28

    Giving agents write access to systems like Postgres, Kubernetes, AWS, and ClickHouse is powerful for incident response but highly dangerous. Agents can be prompt-injected through support systems, allowing them to perform nefarious actions (e.g., running `psql` subprocesses to delete tables).

  2. Security Must Be Externalized 4:07

    The speaker asserts that security cannot be 'wishful thinking' based on model obedience. Since agents are untrusted software, the guard mechanism must sit outside of the agent itself.

  3. Claw Patrol: Byte-Level Inspection 6:45

    Claw Patrol is a proxy that sits in front of agents and parses every byte leaving them, operating below the HTTP layer. This allows it to detect dangerous paths that are not standard HTTP requests, such as a subprocess spawning `psql` to connect via an EKS endpoint.

Watch on YouTube Full article

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI thumbnail

· 1:03:26

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

This talk provides a deep dive into context engineering for large language model (LLM) agents, focusing on optimizing performance and cost in real-world applications like an AI tutor. The core finding is that compaction techniques (like summarization) are often detrimental because they invalidate the prompt cache, forcing the system to pay full price for every token. For optimal recall and cost efficiency, retaining the full history—especially when leveraging advanced caching mechanisms—is superior to aggressive context reduction.

Key takeaways

  1. Compaction is a potential trap due to Prompt Caching 22:06

    When using prompt caching (which can make cached tokens up to 50 times cheaper), any transformation or summarization of the context invalidates the cache, forcing full-price token usage. Compaction must shrink the context by more than 50 times just to pay for itself.

  2. Full History Retention Wins on Recall 33:33

    Experiments showed that leaving the full history untouched (the 'full history' preset) provided the best memory recall and overall performance, even though it was the most expensive option. Aggressive compaction techniques significantly degraded factual retrieval.

  3. Hybrid Search is Superior for Retrieval 22:48

    For knowledge base browsing, a hybrid search combining semantic similarity (embedding model) with keyword search (BM25) is necessary to achieve high recall. Pure dense retrieval failed when facts were buried at large context sizes (e.g., 400k tokens), while BM25 maintained 100% accuracy.

Watch on YouTube Full article

The Rise of CaaS: Context-as-a-Service for Agentic AI — Omer Primor, Bright Data thumbnail

· 22:20

The Rise of CaaS: Context-as-a-Service for Agentic AI — Omer Primor, Bright Data

The video analyzes the shift from viewing web data as a simple source of information to treating it as dynamic 'context' for agentic AI. The speaker argues that Context-as-a-Service (CaaS) vendors are emerging to provide structured knowledge graphs, acting as vertical search engines. Critically, he emphasizes that at scale, the cost killer is not initial volume but the *frequency* of repeated queries. For persistent knowledge work, owning and building a custom data pipeline—even if time-consuming—can eventually become more cost-effective than continually renting context from third-party vendors.

Key takeaways

  1. Context Decay: Data is never a snapshot 0:02

    Web data decays quickly (e.g., social content < 1 day; news/finance ~30 days). Therefore, extracting context must be treated as an ongoing process, not a one-time effort [2:43].

  2. The Rise of CaaS for Agents 0:06

    AI agents require structured knowledge beyond what general search provides. CaaS vendors address this by developing and indexing specialized knowledge graphs (vertical search) across multiple data sources, enabling deep reasoning [6:32].

  3. Frequency is the Cost Killer at Scale 0:12

    When performing repeated due diligence or market research, every query costs money, even if nothing has changed. This recurring cost (frequency) eventually surpasses the initial setup cost of building an owned pipeline [12:32].

  4. The Tipping Point for Ownership 0:15

    There is a tipping point where the cumulative cost of repeated context queries makes it economically viable to build and own the data retrieval pipeline in-house, potentially bypassing middleman costs [15:22].

Watch on YouTube Full article

From RL to IRL — Gaurav Mishra, Amazon AGI Lab thumbnail

· 17:46

From RL to IRL — Gaurav Mishra, Amazon AGI Lab

The talk details the transition from Reinforcement Learning (RL) in controlled environments ('the game') to real-world deployment (IRL), where agents face significant complexities. While RL is effective for tasks with verifiable outcomes and multiple solution paths, real life introduces partial observability, irreversible actions, expiring credentials, and adversarial content. To bridge this gap, the speaker proposes a 'flight school' approach: training agents in high-fidelity digital sandboxes that simulate messiness (e.g., layout shifts, pop-ups). System improvements include implementing Process Reward Models (penalizing dangerous steps along the path), utilizing Calibrated Confidence (teaching the agent when to escalate to a user), and building robust 'harnesses' with guardrails for checkpointing, rollback, and risk classification.

Key takeaways

  1. RL vs. IRL: The Core Challenge 9:07

    RL works well in controlled environments where the outcome is verifiable. However, when deployed in real life (IRL), agents encounter partial observability (e.g., DOM missing content baked into images) and irreversible actions, causing failures like account lockouts or redirection to malicious sites.

  2. The 'Flight School' Approach 13:49

    Instead of focusing only on the final outcome (exams), agents must be trained in messy, high-fidelity simulations that model real-world edge cases like slow loads, focus stealing, and random account states. Recovery actions (refresh, backtrack) must become native model capabilities.

  3. System Architecture Improvements

    Robust agent systems require upgrading the 'cockpit' (the harness). This includes adding guardrails for action risk classification, implementing checkpointing and rollback mechanisms, and requiring calibrated confidence to determine when human handoff is necessary.

Watch on YouTube Full article

The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore thumbnail

· 21:38

The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore

This talk details advanced web automation techniques for AI agents to interact with websites that actively resist bots. The core methodology involves treating the agent as a 'meat bag with a mouse,' leveraging the Chrome DevTools Protocol (CDP) through a Command Line Interface (CLI). Automation is structured around a 'Sense, Act, Verify' loop and progresses up a three-rung 'Meatbag Ladder'—moving from simple API calls to simulated human input (jitter, overshoot)—to defeat modern security measures like Cloudflare Turnstile and reCAPTCHA v2.

Key takeaways

  1. CLI vs. MCP for Automation 3:30

    While capability is comparable (both achieved tasks 83% of the time), a CLI approach significantly outperforms an MCP server in speed, reuse, and cost. Specifically, a CLI sequence can be programmed once and reused infinitely without model calls, making it faster and up to 75 times cheaper in token cost than repeated model interaction.

  2. The Sense-Act-Verify Loop 9:07

    Effective automation requires a continuous loop: Sensing (via DOM, accessibility tree, or screenshot), Acting (clicking/typing), and Verifying (checking network traffic or screen state). This process must be repeated until the desired action is confirmed.

  3. The Meatbag Ladder Methodology 10:38

    Web resistance requires escalating techniques. Rung One uses synthetic JavaScript clicks (API-like). Rung Two employs real CDP input events, which the page recognizes as trusted inputs. Rung Three simulates full human behavior, including jitter and deliberate overshoot, to bypass advanced bot detection.

  4. Solving reCAPTCHA v2 18:18

    The final challenge (reCAPTCHA v2) is solved by splitting the work: deterministic code handles the rapid, repeatable actions (trusted clicks, iframe piercing), while a specialized AI agent provides the 'eyes and brain' to interpret visual challenges (e.g., identifying specific objects in a grid).

Watch on YouTube Full article

Bringing agents onto the world wide web — Paul Klein IV, Browserbase thumbnail

· 18:26

Bringing agents onto the world wide web — Paul Klein IV, Browserbase

The talk argues that the current bottleneck for building reliable web agents is not model capability (AI), but rather the missing engineering infrastructure—the 'harness' and scalable systems. Reliable browser agents must be multimodal, harness-engineered with memory/skills, and operate on consistent, robust infrastructure. The speaker emphasizes that solving this overhang is an engineering problem that companies can tackle today, accelerating AI diffusion into real-world enterprise use cases (e.g., logistics, banking).

Key takeaways

  1. The bottleneck is the harness, not the model. 7:12

    Model capabilities have advanced significantly; therefore, agents struggle due to missing scaffolding and systems around the LLM (the 'harness'). Building a custom harness optimized for a domain can improve results beyond baseline models. (4:32)

  2. Reliable browser agents require three components. 11:45

    They must be multimodal, harness-engineered (using skills and memory), and run on reliable infrastructure that guarantees consistent page rendering across all runs. (7:05)

  3. The web needs to adapt for agents. 17:04

    Key improvements required from the web include better accessibility trees, adoption of Chrome's Web MCP, solving agent login/authentication securely, and establishing a trust framework (a 'Verisign moment') for agents. (10:24)

  4. Infrastructure must be scalable and consistent. 15:09

    Production environments require infrastructure that can handle thousands of agents, maintain consistency (e.g., preventing mobile/desktop layout shifts), and ideally achieve SOC 2 compliance at scale. (9:09)

Watch on YouTube Full article

Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain thumbnail

· 20:02

Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

The continuous improvement of autonomous agents requires shifting focus from code determinism to data mining agent traces. The speaker argues that observability and continual learning are fundamentally linked: an agent's actions in an environment generate a trace record that serves as the substrate for all future improvements. Techniques like harness engineering, distillation (SFT), and analyzing counterfactual traces allow developers to systematically improve agents at lower costs than relying solely on frontier models.

Key takeaways

  1. Shipping is the First Step

    To gather data for improvement, an agent must be deployed into a real-world environment (shipping it). This process generates valuable trace data from tool calls, API usage, and CLIs.

  2. Observability = Continual Learning 4:04

    There is a tight coupling between observability and continual learning for agents. Both require comprehensive traces—the record of actions taken in the environment—to allow the agent to update its internal knowledge or definition.

  3. The Value of Traces 6:00

    Traces capture fine-grained behavior that simple pass/fail benchmarks miss. They are crucial for proving counterfactuals (e.g., comparing GPT 5.5 vs. GLM 5.2) and understanding how agents behave at a granular level.

  4. Improvement Loop Strategy 13:00

    For agent improvement, the recommended loop is: Start with Harness Engineering (fast feedback, ~2 minutes) $\rightarrow$ Saturate this ceiling $\rightarrow$ Fine-tune the model to break through it $\rightarrow$ Return to Harness Engineering.

  5. The Future of Data 17:00

    Agent activity will generate data at an exponential rate, potentially eclipsing all human-produced data in history. Managing this requires building systems that can efficiently mine and process traces.

Watch on YouTube Full article

Lessons from Studying Every Memory System — Shlok Khemani, Independent thumbnail

· 19:31

Lessons from Studying Every Memory System — Shlok Khemani, Independent

The talk provides a deep dive into the evolution and architectural differences of memory systems in consumer AI applications (ChatGPT, Claude, Gemini). The core thesis is that 'memory' is not a standardized technology but rather a function of compute, requiring careful trade-offs between profile size, update frequency, and context window cost. Speakers highlight that while general architectures are converging toward running profiles, the specific implementation details remain unique to each product, meaning memory cannot be outsourced.

Key takeaways

  1. Memory is not standardized

    There is no single way to implement AI memory; products evolve independently (e.g., ChatGPT uses dense keywords/running profile; Claude uses full sentences/tools).

  2. The Compute Trade-off

    A running profile requires balancing two costs: the cost to maintain (update frequency/compute) and the serving cost (profile length in context window). This trade-off dictates product design.

  3. Memory is a Product Problem

    The biggest limitation of current AI memory systems is not technology, but product design. They often fail to reason over rich external sources like emails or calendars, leading to conflicts (e.g., conflicting travel dates).

Watch on YouTube Full article