The Video Signal

The Video Signal technical video digests

Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind thumbnail

· 18:28

Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind

The presentation details the evolution of LLM agents, demonstrating a shift from complex, brittle Python code loops to declarative, file-based definitions using system instructions and skills. The speaker shows that modern agent architectures, such as the Gemini API's anti-gravity agent, utilize a hosted sandbox and network proxy to manage state and credentials securely. This allows agents to operate using general-purpose tools (like GitHub CLI or Google Search) defined in files (e.g., `AGENTS.md`), drastically reducing the need for thousands of lines of custom orchestration code.

Key takeaways

  1. The Agent Evolution: Code to Files

    Agent development is moving away from writing explicit Python loops, JSON schemas, and tool routing logic. The core functionality is now expressed in files (Markdown/Skills) that define instructions, rules, and capabilities, allowing the model to use general tools.

  2. Server-Side State Management 14:02

    The new architecture handles complex tasks by moving loops, tool routing, session state, and context compaction to the server side, requiring only a single API call with new inputs.

  3. Security and Isolation 12:32

    A hosted sandbox and network proxy ensure that the agent never sees the actual credentials, injecting tokens only when outbound requests are made, and allowing domain restriction for enhanced security.

  4. Focus on Domain Logic 16:40

    The primary work for developers is now defining the domain instructions, rules, and evaluation criteria (the 'what'), rather than writing the infrastructure code (the 'how').

Watch on YouTube Full article

How We Solved Agent Building — Andrew Qu, Vercel thumbnail

· 17:34

How We Solved Agent Building — Andrew Qu, Vercel

Andrew Qu details Vercel's journey in solving agent building, evolving from initial manual data querying to complex, multi-step agent architectures. The major breakthrough was realizing that the core capability was not a clever toolset, but the simple file system (list, read, bash). This insight led to the development of Eve, a framework designed to bring file system conventions (like Next.js) to agents, enabling developers to build highly customized, production-ready agents with minimal effort.

Key takeaways

  1. The File System Unlock 11:57

    The most significant breakthrough was realizing that powerful agents could be built using a minimal set of tools—specifically, file system operations (list, read, run bash)—which models are already well-trained on. This approach allowed agents to explore and write where needed, leading to a massive increase in performance (the eval score basically doubled).

  2. Focus on Company-Specific Knowledge 17:00

    While off-the-shelf agents are useful, the speaker argues that the most effective agents are those stuffed with deep, company-specific knowledge. These custom agents provide the most 'juice' and save significant time across various departments (e.g., legal, finance, data science).

  3. The Need for a Framework (Eve) 15:40

    To simplify agent development, Vercel built Eve, described as the 'Next.js for agents.' It adopts file system conventions, allowing developers to easily declare skills, tools, and channels, making the process of building and iterating on agents highly manageable.

Watch on YouTube Full article

No Memory, No Harness: Why the Database Is the Last Line of Defense — Kay Malcolm, Oracle thumbnail

· 21:37

No Memory, No Harness: Why the Database Is the Last Line of Defense — Kay Malcolm, Oracle

Kay Malcolm argues that while AI models (agents) are powerful, they are incomplete without a robust, centralized memory system. She frames the agent as the 'brain,' the surrounding system as the 'harness' (body), and the database as the 'central nervous system' (memory). The core problem addressed is that current systems (like Git) track code changes, not the human intent or context behind them. To solve this, she advocates for using a unified Oracle AI database to store all five types of agent memory (short-term, long-term, episodic, procedural, and semantic) in a single source of truth, preventing context loss when scaling to enterprise teams.

Key takeaways

  1. AI's Limitation: Code vs. Intent 5:36

    AI tools make individuals faster, but they do not automatically make teams more productive if the context and reasoning behind the code are not shared. Git only records what changed, not the human intent (3:36).

  2. The Agent Architecture Model 11:54

    An enterprise agent requires three components: the Model (the brain), the Harness (the body, enabling action), and Memory (the central nervous system, carrying context) (7:14).

  3. The Five Types of Memory 13:29

    Effective agent memory must distinguish between: short-term (within a session), long-term (across sessions), episodic (what happened last time), procedural (steps taken), and semantic (meaning) (8:09).

  4. The Need for a Single Source of Truth 17:34

    When data is spread across multiple specialized databases (relational, document, graph, vector), agents struggle to reconcile the truth, often guessing incorrectly and wasting tokens (10:54).

  5. The Solution: Unified Database Memory

    A unified database (like the Oracle AI database) is necessary to store all memory types (JSON, relational, graph, vector) in one place, ensuring the agent's memory is non-negotiable and accessible across the entire team (14:24).

Watch on YouTube Full article

Share your Managed Deep Agent with your team using Slack thumbnail

· 5:04

Share your Managed Deep Agent with your team using Slack

This guide details the process of deploying a Managed Deep Agent, initially developed in LangSmith Studio, to a production environment using Slack as the primary interaction layer. The process involves running `slack init` to configure the connection, executing a deployment, completing the Slack authorization step, and finally customizing the agent's appearance and message trigger logic for optimal team integration.

Key takeaways

  1. Agent Deployment Workflow

    To connect an existing agent to Slack, run the `slack init` command, followed by a redeployment. The first deployment requires completing a Slack authorization step to link the agent to the internal workspace.

  2. Monitoring and Tracing 0:01

    When the agent is live, all user interactions (requests) are logged and traceable within the LangSmith dashboard, allowing engineers to monitor complete request traces even though the user only sees the final answer in Slack.

  3. Agent Customization 0:02

    The agent's name, description, icon, and background color can be customized by modifying the setup code in the channels directory and redeploying the agent.

  4. Message Trigger Configuration 0:03

    Two key trigger options exist: manual tagging (default) or 'trigger on all messages' (setting `trigger_on_all_messages` to true). Enabling the latter allows the agent to respond to any message in a dedicated channel, not just those directly addressed to it.

Watch on YouTube Full article

We let an AI agent execute Bash and lived to talk about it — Sarah Sanders, PostHog thumbnail

· 21:00

We let an AI agent execute Bash and lived to talk about it — Sarah Sanders, PostHog

The talk details the architecture and security challenges of building agentic CLI tools, exemplified by PostHog's 'Wizard.' The speaker emphasizes that an agent with command execution capabilities is essentially a 'malware starter pack.' Security must be layered, deterministic, and must account for dangerous inputs originating not only from users but also from the system's own supply chain (the 'Context Mill'). The solution involves implementing a dedicated, deterministic scanner (Warlock) and ensuring that the LLM layer acts only as an 'adviser,' never an enforcer.

Key takeaways

  1. The Danger of Agentic Tools 0:03

    An agent that can run commands is structurally similar to malware, requiring extreme caution. The threat model must account for how 'attacks compose' from multiple, seemingly innocent vulnerabilities.

  2. Context Supply Chain Risk 0:11

    The most dangerous input can come from the system's own supply chain (the Context Mill), such as a seemingly harmless prompt injection payload signed by the development team.

  3. Security Must Be Deterministic 0:15

    For enforcement, security mechanisms must be deterministic and mechanical. The LLM layer should only be an 'adviser' to reduce noise, never the primary enforcer, to prevent failure modes.

Watch on YouTube Full article

Harness Engineering: Building the Production Cage for Powerful Domain Agents — Mike Chambers, AWS thumbnail

· 20:46

Harness Engineering: Building the Production Cage for Powerful Domain Agents — Mike Chambers, AWS

The presentation introduces 'Harness Engineering,' a critical concept for building production-grade AI agents at scale. Mike Chambers distinguishes between agents that are used (e.g., coding assistants) and agents that are built. For built agents, the harness encompasses all non-model components—such as memory, skills, tools, identity, and context management—that must scale independently. The core principle is that scaling these components separately, rather than deploying them in a single container, is essential for handling thousands of users and maintaining reliability.

Key takeaways

  1. Two Types of Agents 4:05

    Agents are categorized into 'agents we use' (productivity tools, coding assistants) and 'agents we build' (production-scale systems). The approach for built agents requires careful architectural planning.

  2. Defining the Harness 7:04

    A harness is defined by subtraction: take an agent and remove the model component; everything left over is the harness. This includes the infrastructure, skills, and tools.

  3. Scaling Built Agents 10:57

    For production agents, the harness must manage complex concerns like loop management, scaling, payments, identity, runtime, context management, and observability. Attempting to containerize everything together is incorrect for high scale.

  4. Avoiding 'Slop Ops' 10:07

    Build engineers must avoid 'slop ops' (clicking around a console to deploy resources). Instead, agents must build infrastructure using Infrastructure as Code (IaC) to maintain ownership and control over cloud deployments.

Watch on YouTube Full article

Sam Altman and Apple's New CEO are Fighting Over One Thing. It's Not What You Think. thumbnail

· 29:26

Sam Altman and Apple's New CEO are Fighting Over One Thing. It's Not What You Think.

The core conflict between Apple and OpenAI is not hardware, but control over the user's 'working context' and personal data history. Apple is strategically positioning its devices (e.g., iPhone 18 Pro, iPhone Duo) and new chips (A20 Pro) to keep the user's computing life centered on its ecosystem. However, the video argues that AI agents (like ChatGPT or Claude) are building value by owning the user's workflow, creating a threat that Apple must counter by monetizing expanded AI access through subscription fees. Google and Nvidia are positioned as key suppliers, creating a complex competitive landscape where the battle is for the user's trust.

Key takeaways

  1. The Battleground is the Working Context

    The primary contest is over who holds the user's accumulated history and 'working context,' as this relationship is the most valuable asset in the long term. Whoever controls this data stream dictates the recurring revenue stream.

  2. Monetizing Intelligence Access 2:00

    Apple's announcement includes a critical line: 'Certain features that rely on server side models will have daily usage limits. All of those will be available with an additional fee for expanded access.' This suggests a model where the hardware purchase is complemented by a recurring AI subscription fee.

  3. Hybrid AI Architecture 7:40

    Apple is adopting a hybrid solution: local compute (on-device processing for privacy, e.g., local speech) handles simple interactions, while complex requests are routed to the cloud. This maintains the device's centrality while leveraging external AI power.

  4. Competitive Pressure from Agents 16:40

    OpenAI and Google are building agents that operate across the entire stack (hardware, software, and cloud), betting that the value of the agent's work (e.g., planning a trip) will justify a recurring payment, regardless of the device used.

Watch on YouTube Full article

Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic thumbnail

· 13:21

Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic

The presentation challenges the assumption that all tokens used in agentic systems are fungible. Instead of relying solely on increasing the token budget (brute force), the speakers propose giving tokens specific 'jobs' through advanced strategies. These strategies—Advising, Grading, and Dreaming—allow agents to improve performance and reliability within a fixed budget, particularly when the task requires high precision (e.g., financial analysis). The analysis showed that while increasing the budget improves performance across all methods, specialized strategies significantly reduce the true cost (tokens) required to achieve a perfect, reliable outcome.

Key takeaways

  1. Tokens are not fungible

    The core thesis is that tokens should be assigned specific jobs (e.g., advising, grading, dreaming) rather than being used indiscriminately for simple execution. This allows for more efficient use of computational resources.

  2. Three core agent strategies 3:45

    Strategies include: 1) Advising (splitting an executor from an adviser that provides mid-task guidance); 2) Grading (defining a rubric and using a grader to iterate until a perfect outcome is achieved); and 3) Dreaming (allowing tokens to inspect transcripts and write findings to memory for future runs).

  3. Cost vs. Accuracy in Financial Analysis 11:40

    When evaluating financial tasks (like creating a P&L statement), 80% accuracy is considered useless if 100% accuracy is required. The analysis showed that the baseline 'Execute' strategy required an average of 1.8 million tokens to achieve a perfect answer, while 'Advise' and 'Grade' were significantly more token-efficient.

  4. Optimization depends on the goal

    The choice of strategy depends on the optimization goal: choose 'Advise' for maximum token efficiency, or 'Grade'/'Dream' if maximizing the percentage of perfect, reliable runs is the priority.

Watch on YouTube Full article

Agentic Engineering Benchmarks: How I RANK Astra, Fable 5.1, and Open-Weights thumbnail

· 39:23

Agentic Engineering Benchmarks: How I RANK Astra, Fable 5.1, and Open-Weights

The video argues that relying on generalized AI benchmarks (like the Artificial Analysis Index) is misleading because model performance is a complex, three-dimensional problem involving performance, cost, and speed. The speaker advocates for selecting a curated set of five specialized benchmarks—Terminal-Bench, APEX Agents, AutomationBench, AA-Omniscience, and DeepSWE—to accurately assess model capabilities for specific, real-world agentic engineering tasks, rather than relying on single aggregate scores.

Key takeaways

  1. Model Selection is a 3D Problem 0:40

    Choosing a model requires balancing performance, cost, and speed simultaneously. A model that performs best may be prohibitively expensive or slow, making the trade-off critical for production systems.

  2. Focus on Variance, Not Indices 1:50

    The most valuable benchmarks show significant variance in model performance, indicating 'alpha' or information gain. Flat-lining benchmarks (like some long context retrieval tests) are considered saturated and uninformative.

  3. The Importance of Guardrails (Alignment) 10:30

    AutomationBench is highlighted for its focus on guardrail adherence. A model must not only complete the objective but also avoid triggering guardrail violations, which is crucial for reliable, production-grade agents.

  4. The Value of Honesty (AA-Omniscience) 17:30

    The AA-Omniscience benchmark measures hallucination rate and the cost of honesty. It is critical that an agent can 'opt to not answer' when lacking information, as this prevents cascading errors in long-running agent pipelines.

  5. Long-Horizon Work is Key 27:30

    DeepSWE v1.1 is emphasized for testing long-horizon software engineering tasks. The goal is to build systems that operate autonomously with minimal human oversight, requiring models that maintain consistency over extended periods.

Watch on YouTube Full article

How Developers Secure AI-Generated Code: 5 Security Best Practices thumbnail

· 11:28

How Developers Secure AI-Generated Code: 5 Security Best Practices

As AI accelerates software development, traditional security practices designed for human-written code are insufficient. This talk outlines five critical 'shift-left' security principles necessary for building trust in AI-assisted code. The focus shifts from merely reviewing code to validating the outcome, rigorously managing dependencies, and ensuring security is an ongoing, continuous process across the entire development lifecycle.

Key takeaways

  1. Trust the Outcome, Not Just the Generation 2:20

    AI-generated code can compile and pass tests while still harboring unseen security risks (e.g., unauthorized data leaks, failure to fail safe). Validation must focus on the system's behavior and expected results under real-world conditions, not just technical functionality.

  2. Security Must Start During Development 3:25

    Integrating security early (shifting left) is crucial. This involves automatically running static source analysis, dynamic penetration testing, and secret scanning *while* the code is being written, rather than treating it as a final checkpoint.

  3. Validate Generated Dependencies 5:30

    AI introduces new dependencies (packages, libraries, services) that carry inherent risk. Developers must scrutinize these dependencies for package reputation, vulnerabilities, licensing, and source integrity, as security incidents often originate in the software supply chain.

  4. Consider Intent Over Code Quality 6:50

    The solution must address the business intent, not just the technical requirements. A code flow may be elegant but still violate security policies if the underlying business rules or access controls are misunderstood or improperly defined.

  5. Security is an Ongoing Practice 8:00

    Security validation must be continuous, extending far beyond initial deployment. The process must incorporate continuous monitoring, vulnerability detection, dependency patching, and policy enforcement throughout the entire 'develop, test, deploy, monitor, improve' loop.

Watch on YouTube Full article

Build A Reasoning Model Scratch 3: The Verifier for Evaluation and RL with Verifiable Rewards thumbnail

· 1:26:47

Build A Reasoning Model Scratch 3: The Verifier for Evaluation and RL with Verifiable Rewards

This video details the implementation of a robust verifier pipeline for evaluating Large Language Models (LLMs) on mathematical benchmarks, specifically using the MATH-500 dataset. The verifier is crucial for establishing a baseline performance metric, which will later be used in Reinforcement Learning with Verifiable Rewards (RLVR) training. The process involves eight key steps: generating text, extracting the final answer (ideally from a boxed format), normalizing the answer to a canonical form, mathematically verifying its equivalence to a ground truth using `sympy`, and finally grading the answer to compute overall model accuracy.

Key takeaways

  1. LLM Evaluation Methodologies 2:01

    Model evaluation can be categorized into Multiple Choice, Verifier-based, Leaderboard-based, and LLM Judge methods. The verifier approach is preferred here because it provides an objective evaluation with a hard ground truth answer, which is necessary for RLVR training.

  2. The Verifier Pipeline 19:13

    The evaluation pipeline is complex, requiring steps to extract the final answer, normalize the format (e.g., removing LaTeX fluff), verify mathematical equivalence, and grade the result. This robustness is critical for reliable benchmarking.

  3. Model Performance Comparison 27:13

    The base model's accuracy (e.g., 15.6% on MATH-500) is significantly lower than the reasoning model's accuracy (e.g., 50.8%). This highlights the value of specialized reasoning techniques and training.

  4. Reproducibility Caveats 20:15

    Model evaluation results can vary based on the computing device (CPU, MPS, CUDA) and due to floating-point math, necessitating running evaluations multiple times and averaging the results for robustness.

Watch on YouTube Full article

The US–China AI Arms Race Isn't Real But The Lobbying Is. My Guest Worked Both Sides. thumbnail

· 48:24

The US–China AI Arms Race Isn't Real But The Lobbying Is. My Guest Worked Both Sides.

The discussion argues that the framing of the US-China AI competition as a zero-sum 'arms race' is a core misconception (02:54). Instead, the intelligence generated by AI is becoming 'ambient' and commoditized through open-source, open-weight models, making it impossible to monopolize. The future value of AI will shift from building the largest models (e.g., 10 trillion parameter models) to applying specialized, smaller models (e.g., 10 billion parameter models) for specific societal betterment, such as drug discovery or infrastructure improvement. The ultimate opportunity lies in leveraging AI's productivity gains to foster global cooperation and human-centric activities, rather than military competition.

Key takeaways

  1. The AI Arms Race is a Misconception 1:31

    The belief that AI is a zero-sum contest with a single winner is flawed. The commoditization of high-quality intelligence via open-source and open-weight models means that intelligence is becoming ambient and impossible to hoard, much like electricity was (09:01).

  2. Focus on Specialization, Not Scale 16:49

    The misconception is that the race is to build the biggest, most super AI model. The more efficient approach is to use smaller, specialized models (e.g., a 10 billion parameter model) tailored for specific tasks, allowing for better resource allocation into societal needs like hospitals and infrastructure (16:49).

  3. The Future Requires High-Dimensional Skills 18:30

    Young people should focus on developing a broad, T-shaped experience set—combining wide reading (history, philosophy, sociology) with deep, end-to-end technical skills (designing, building, deploying, and sunsetting a system). Specialization alone is insufficient because AI can now provide answers that require critical judgment to validate (11:10).

  4. Cooperation is the Path Forward 33:20

    The most likely path is not conflict, but a global shift toward cooperation, modeled after historical events like the Marshall Plan. Redirecting spending from military spending to global development (especially the Global South) is necessary to stabilize the economy and prevent a collapse (2000).

Watch on YouTube Full article

Essential Skills for Becoming an AI Engineer: RAG, AI Agents, & More thumbnail

· 10:55

Essential Skills for Becoming an AI Engineer: RAG, AI Agents, & More

The video outlines the modern skill stack required for an AI Engineer, defining the role as building and deploying systems around existing Large Language Models (LLMs), rather than researching foundational models. The necessary skills are structured into three critical tiers: foundational engineering skills (Python, Git, APIs), specialized AI techniques (RAG, embeddings, agents), and deployment/operations expertise (Containerization, Observability). The speaker emphasizes that mastering this structured approach is crucial for building reliable, production-grade AI solutions.

Key takeaways

  1. AI Engineer vs. ML Researcher

    An AI Engineer builds applications using existing models (e.g., Frontier, Open Source), connecting them to data and tools. An ML Researcher focuses on training foundational models from scratch and publishing new architectures.

  2. The Three-Tier Skill Stack 2:00

    The required skills are structured in three tiers: 1) Foundational skills (Python, Git, APIs); 2) AI-specific skills (Embeddings, RAG, Agents); and 3) Deployment/Ops skills (Containerization, Observability).

  3. The Importance of Grounding Data (RAG) 4:00

    Retrieval Augmented Generation (RAG) is critical for providing LLMs with accurate, company-specific information (e.g., policies, legal documents) to prevent hallucination. This involves chunking documents, embedding them into vectors, and feeding relevant context into the LLM's context window.

  4. AI Agents and Tool Use 5:20

    AI Agents represent a significant advancement, allowing systems to dynamically decide next steps, call external tools, observe results, and iterate in a loop, moving beyond simple predefined workflows.

Watch on YouTube Full article

Trying the new Claude Eval tool thumbnail

· 58:39

Trying the new Claude Eval tool

The video provides a technical deep dive into the new Claude Evals tool, designed to evaluate the performance of AI plugins and skills. While the tool automates test case generation and scoring (comparing runs with and without the plugin), the speakers highlight significant limitations. The process is currently highly dependent on manual, pre-collected data (e.g., defining failure modes, providing positive/negative examples) and lacks an integrated, 'in-the-loop' annotation interface, making its real-world application challenging for robust build pipelines.

Key takeaways

  1. Claude Plugin Eval Functionality 2:05

    The `Claude plugin eval` tool allows users to evaluate plugins by generating test cases, running the plugin against them, scoring the results, and comparing the scores with and without the plugin's intervention. This process generates a terminal output and an HTML report.

  2. Evaluation Input Requirements 10:10

    The tool requires users to define quality standards by providing examples of 'good' and 'bad' outputs, identifying specific failure modes (e.g., overediting, abrupt transitions, jargon), and supplying real prompts or past outputs.

  3. Process Limitations 31:40

    The current chat-based interface is criticized for being 'out of the loop.' Effective evaluation requires continuous, in-situ feedback and annotation, which the current tool does not adequately support.

Watch on YouTube Full article

Building ambitious software — Jonathan Kelley, Dioxus Labs & Cognition thumbnail

· 19:14

Building ambitious software — Jonathan Kelley, Dioxus Labs & Cognition

Jonathan Kelley discusses building ambitious, cross-platform software using Rust (Dioxus), detailing the challenges of maintaining high quality across a complex stack. He highlights how AI coding agents have dramatically accelerated development—allowing the team to implement features like deeply integrated Kotlin and Swift plugins in weeks—but notes that this shift changes the focus from writing code to maintaining architectural quality. The core message is that while code is now cheap, architectural integrity and quality remain the primary concerns for ambitious projects.

Key takeaways

  1. The Shift in Software Engineering Focus 17:12

    With AI agents handling the low-level development burden (e.g., fighting the borrow checker), the job of a software engineer shifts from writing code to architecting elegant solutions and thinking about system evolution.

  2. Code Quality vs. Code Quantity 17:12

    The speaker asserts that 'code is cheap now, but quality is not.' Maintaining a high quality bar across the entire codebase is critical for developer experience and business stability.

  3. AI Agents as Knowledge Accelerators 13:20

    Coding agents excel at 'knowledge problems,' such as sifting through thousands of pages of documentation, reverse engineering APIs, and recalling deep technical specifications (e.g., CSS specs), accelerating tasks that previously took years of hand-written effort.

Watch on YouTube Full article

Manage your agents while you’re on the move with the Antigravity Remote Control thumbnail

· 0:48

Manage your agents while you’re on the move with the Antigravity Remote Control

Antigravity Remote Control allows users to manage and monitor long-running agents across multiple machines from a single, unified interface (browser or app). This capability ensures that local context is retained, eliminating the need to recreate or sync the build environment when working on different devices.

Key takeaways

  1. Centralized Agent Management

    Users can control and monitor all running agents from one place, providing a unified view for managing development sessions.

  2. Context Persistence

    All local context is retained, meaning the build environment does not need to be recreated or synced when switching devices.

  3. Workflow Integration

    The system supports push notifications for review changes (diffs) and allows users to answer questions and maintain progress from any location.

Watch on YouTube Full article

Lecture 115: Proving Kernels Correct Instead of Testing Them thumbnail

· 40:07

Lecture 115: Proving Kernels Correct Instead of Testing Them

This lecture details the necessity and methodology of formally verifying GPU kernels, moving beyond traditional testing due to the massive input space and non-deterministic nature of GPU execution. The core approach involves reasoning about kernel correctness at the PTX level using Satisfiability Modulo Theories (SMT) solvers. The process requires modeling mathematical concepts (like floating-point operations and unbounded integers) as abstract 'reals' to prove equivalence between a candidate kernel and a reference implementation, thereby guaranteeing algorithmic correctness.

Key takeaways

  1. Limitations of Testing GPU Kernels 1:30

    Due to the vast input space and the non-deterministic scheduling inherent in GPU hardware (e.g., Nvidia), testing alone cannot guarantee correctness. Errors can occur across various boundaries (e.g., denormal boundaries, cancellation terms), and the execution order cannot be guaranteed at runtime. (0:01:30)

  2. Formal Verification at the PTX Level 2:10

    To prove correctness, the goal is to show that for every value X, the candidate kernel produces the same output as the reference kernel. This verification is performed at the PTX level because it provides relatively well-defined semantics, allowing the system to cover inputs from various sources (Triton, CUDA, inline PTX). (0:02:10)

  3. SMT Solvers and Mathematical Abstraction 3:20

    Formal verification uses SMT solvers to convert code into mathematical boolean abstractions. By modeling values as 'reals' (abstract values with infinite precision) and unbounded mathematical integers, the system can prove algorithmic equivalence without requiring bit-exact checks, which would lead to an unmanageable 'explosion' of clauses. (0:03:20)

  4. Advanced Verification Boundaries 4:30

    Future work focuses on formally verifying complex boundaries, including sandboxes (to prevent escape paths from LLM-generated code) and numerical stability (e.g., quantization stability), which are addressed separately from algorithmic correctness. (0:04:30)

Watch on YouTube Full article

Proving Kernels Correct Instead of Testing Them thumbnail

· 44:41

Proving Kernels Correct Instead of Testing Them

The talk advocates for formally proving the correctness of GPU kernels instead of relying solely on testing, especially for AI-generated code. This approach is necessary because AI agents can engage in 'reward hacking' and exploit non-deterministic hardware behaviors. Formal verification is performed at the PTX level, utilizing SMT solvers and advanced proof assistants like Lean to model hardware semantics, ensuring algorithmic correctness and numerical stability.

Key takeaways

  1. Formal Verification vs. Testing 2:00

    Testing cannot cover the entire problem space, and GPU execution is non-deterministic, meaning a kernel might behave differently in production than in a test environment. Formal verification proves that for every input X, the candidate kernel produces the same output as the reference kernel.

  2. Addressing Hardware Vulnerabilities 2:30

    Agents can exploit vulnerabilities like 'control hijacking' or 'monkey patching' (e.g., timing functions, numerical precision) within the execution sandbox. Formal verification is required to ensure the integrity of the entire system, including the sandbox and grader.

  3. Modeling Complexity (Reals and PTX) 3:40

    To avoid the computational explosion of bit-exact modeling, the process models values at the SMT level as 'reals' (assuming infinite precision) and performs verification at the PTX level, which provides relatively well-defined semantics across different compiler toolchains (Triton, CUDA, etc.).

  4. Advanced Proof Techniques 5:50

    The process involves converting the PTX kernel into mathematical clauses, using SMT solvers to check for divergence, and leveraging proof assistants like Lean to model hardware semantics and perform arbitrary searches for proofs.

Watch on YouTube Full article

Is Omarchy The Last Desktop You'll Ever Need? thumbnail

· 17:59

Is Omarchy The Last Desktop You'll Ever Need?

The video discusses the shift in operating system control, moving from rigid, centralized defaults to highly customizable, agent-driven environments. While Omarchy (a Linux desktop) exemplifies this future, the core lesson is that AI agents can modify existing Mac and Windows OS settings by interacting with documented, exposed APIs and configuration files. The key to successful agent implementation is scoping changes narrowly, defining specific inputs/outputs, and carefully managing permission levels to maintain system dependability.

Key takeaways

  1. Agent-Driven OS Customization

    Operating systems are fundamentally collections of decisions made by others. AI agents, like those demonstrated by Omarchy, allow users to 'time travel' and modify these decisions to fit highly specific, individual needs, rather than relying on universal defaults.

  2. Scoping Changes and Permissions 5:00

    When requesting changes, users must be highly specific (e.g., 'Find this setting, tell me the file that needs to change') rather than vague (e.g., 'Make it better'). Agents should only request the minimum necessary permissions, and the system must provide mechanisms to test and undo changes.

  3. The Wrapper Pattern

    A useful pattern is to build an agent wrapper around an existing, functional tool (like using Omaport with Arclone). This allows the agent to create a new, personalized experience without needing to reinvent the complex underlying functionality.

  4. Local Control vs. Processing

    It is crucial to distinguish between local control over an application and local processing of data. Users must be mindful that an agent running on a local machine may still send requests to external, cloud-hosted models.

Watch on YouTube Full article

How to set up Herdr for multi-agent coding (full guide) thumbnail

· 28:40

How to set up Herdr for multi-agent coding (full guide)

This guide details setting up Herder, a tool described as the 'meta' for multi-agent coding, allowing developers to manage and orchestrate multiple AI coding agents and projects within a persistent terminal environment. The setup emphasizes creating isolated 'spaces' for different projects, integrating various AI model harnesses (e.g., Grock, Codeex, Claude), and optimizing the workflow using advanced CLI tools like zoxide, Neovim, and Lazygit. The core benefit is achieving a single, persistent interface for complex, multi-agent development cycles, minimizing context switching and session loss.

Key takeaways

  1. Persistent Session Management

    Herder allows developers to maintain persistent terminal sessions (spaces) that survive terminal closures, similar to T-Mix, ensuring continuous work history and state for CLI-based coding tools.

  2. Agent Orchestration and Delegation

    The Herder skill enables agents to communicate and delegate tasks across different sessions and models (e.g., using Claude to review code started by Grock), providing full transparency into the multi-agent workflow.

  3. Customization via config.toml 20:32

    All aspects of Herder—including colors, spacing, and keyboard shortcuts—can be customized using the `config.toml` file, which can even be configured by an AI agent.

  4. Optimizing Navigation and Workflow

    Integrating tools like zoxide for faster folder navigation and Lazygit for Git status review allows the entire development process to be contained within a single, optimized terminal interface.

Watch on YouTube Full article