The Video Signal

Channel

AI Native Dev

Digests from AI Native Dev

Baz, Docker & Meta on Verifying Agent Code thumbnail

· 10:14

Baz, Docker & Meta on Verifying Agent Code

The session discusses the critical limitation of current AI coding agents: while they excel at generating code, they often fail at verifying that the code meets complex specifications. Speakers from Baz, Docker, Meta, and Christopher Batey presented advanced strategies—such as using real systems as test oracles, implementing anti-test slop initiatives, and elevating human review to Architectural Decision Records (ADRs)—to ensure reliability and maintain system integrity when integrating AI-generated code.

Key takeaways

  1. Verification is the Hard Part

    The primary challenge with coding agents is not writing code, but proving that the generated code accurately implements the required specification. Agents are optimized for feature generation, not full spec extraction or verification.

  2. Specifying Bugs, Not Features

    Baz demonstrated that providing explicit evidence of a bug (e.g., a screen recording of an overlapping button) is far more effective than general specs, as agents may fail to replicate specific, subtle behavioral constraints.

  3. Test Oracles Over 100% Coverage

    Attempting to achieve 100% test coverage is inefficient. Instead, pointing tests at a real, stable system (a 'test oracle,' like S3) and running a limited set of critical tests can effectively lock down behavior and validate system interactions.

  4. Anti Test Slop Initiative

    Meta implemented an anti test slop initiative, using a separate AI tool to judge the quality and necessity of AI-generated tests, thereby reducing CI overhead from useless or redundant tests.

  5. Elevating Human Review to ADRs

    When reviewing large pull requests (e.g., 7,000 lines), human review must move earlier into the process. Architectural Decision Records (ADRs) should capture system-level decisions, allowing agents to check implementations against a structured, human-reviewed architectural baseline.

Watch on YouTube Full article

Tessl Code Review: Catching 74% of Bugs Pre-Merge thumbnail

· 25:56

Tessl Code Review: Catching 74% of Bugs Pre-Merge

Tessl Code Review is an AI-native code reviewer designed to address the critical bottleneck of trusting code generated by coding agents in a software factory. It functions as an automated GitHub app that runs specialized, customizable review rules (lenses) against Pull Requests (PRs). The system emphasizes self-improvement loops, allowing the review process to learn from the codebase and the team's specific best practices, thereby increasing defect detection rates (claiming up to 74% detection).

Key takeaways

  1. AI Code Review and the Software Factory 5:56

    Building a 'software factory' requires more than just plumbing; it requires self-improvement loops that continuously refine the context and skills used by coding agents. Code review is positioned as a critical loop that feeds back into the factory to improve code quality before merging.

  2. Customization via Lenses and Globs 22:07

    The system uses customizable 'lenses' (review rules) that can be tuned to specific domains (e.g., audio DSP) and restricted to specific file paths using globs. This allows for highly targeted reviews, even in large mono-repos.

  3. Local and CI Integration 23:50

    Tessl Code Review can be run directly via the CLI (`Tessl code review give it the profile`) against local changes, enabling developers to catch defects before opening a PR, complementing its automated GitHub app integration.

Watch on YouTube Full article

Meta, Stanford & Odevo on Agentic Coding at Scale thumbnail

· 10:10

Meta, Stanford & Odevo on Agentic Coding at Scale

The session explores scaling agentic coding adoption from a single team to hundreds of engineers. Key findings highlight that while AI tooling can drive massive organic community growth (e.g., Meta reaching 80%+ weekly usage), success is highly dependent on organizational maturity. Speakers warn that deploying agents into an organization with weak software delivery practices will worsen outcomes, emphasizing that foundational improvements—such as robust CI/CD pipelines, dedicated platforms, comprehensive testing, and established coding standards—must precede advanced AI adoption.

Key takeaways

  1. Meta's Adoption Strategy 1:19

    Meta grew an organic community from ad hoc usage to over 40 times its original size. Weekly tool usage increased from under half to above 80%, demonstrating that sustained adoption can be achieved without mandatory enforcement. (00:01:39)

  2. Performance Spread and the 10x Engineer 2:47

    Studies across 150,000 engineers show the widest performance spread ever measured. Contrary to initial hypotheses, top performance is now being achieved by individuals skilled in creating and utilizing agents. (00:02:47)

  3. Prerequisites for Agentic Coding 5:11

    The 2025 DORA report warns that pointing agents at an organization already struggling with software delivery will make things worse. Successful adoption requires fixing fundamentals first: CI/CD, a platform, tests, and coding standards. (00:05:1)

Watch on YouTube Full article

Google, McKinsey & Dave Farley on AI Code Review thumbnail

· 11:15

Google, McKinsey & Dave Farley on AI Code Review

The rapid acceleration of AI code generation creates a critical bottleneck in traditional software development processes, specifically the code review phase. Speakers discuss how the current ratio—where generating code is fast (seconds) but reviewing it remains slow (hours)—is breaking down established social contracts and trust within open-source collaboration. Solutions proposed include shifting focus from quality control to engaging with mental models, implementing advanced agent memory systems (like Tapes), and prioritizing precise, prescriptive language over vague natural language.

Key takeaways

  1. The Code Review Bottleneck 0:26

    Current processes assume a slow ratio: generating code takes 30 seconds, but reviewing it can take an hour. This mismatch strains maintainers, especially when agents generate multiple pull requests (PRs) rapidly, leading to 'trust breakdown' regarding who actually reviewed the code.

  2. Reviewing Mental Models 5:18

    Code review should not be viewed as quality control. Instead, it is a mechanism for engaging with and aligning multiple mental models (the developer’s model, the system’s model, and colleagues' models). Maladaptive creativity can cause these three models to diverge.

  3. Agent Hallucination & Memory 7:33

    AI agents can 'politely hallucinate,' reporting progress without actual learning or completion. To combat this, advanced systems must capture raw agent sessions and process them into structured 'observational memory' for continuous improvement.

  4. Precision over Natural Language 10:02

    Natural language is inherently vague and open to misinterpretation, making it insufficient as the sole programming language of the future. Precision requires prescriptive instructions that go beyond general conversational prompts.

Watch on YouTube Full article

NVIDIA, Docker & Hud on Agents in Production thumbnail

· 10:04

NVIDIA, Docker & Hud on Agents in Production

The discussion explores the operational challenges of deploying AI agents in a production environment (24/7 operation). Key insights emphasize that successful agent deployment requires shifting focus from root cause analysis to comprehensive context and observability. Speakers covered topics including using agents with combined data sources (Elastic logs + ServiceNow), redesigning automated fixes for human consumption, optimizing GPU utilization during tool calls, and leveraging AI-built tracing frameworks for debugging rare bugs.

Key takeaways

  1. Context over Root Cause Analysis 2:10

    When agents are running 24/7 in production, the most critical resource is context—understanding what changed yesterday and the relationships between services. This proactive data knowledge is more valuable than traditional root cause analysis.

  2. Automated Fixes Must Convince Humans 5:40

    Simply automating investigations and opening pull requests (PRs) for high-impact fixes is insufficient, as developers often ignore them. The output must be rebuilt to convince the human developer of its value and priority.

  3. GPU Idle Time During Tool Calls 7:10

    A counterintuitive finding is that when an agent makes a tool call, the GPU sits idle. Properly accounting for this CPU-intensive period allows users to serve roughly twice as many users compared to benchmark predictions that ignore tool calls.

  4. AI-Built Tracing Frameworks 9:00

    For debugging rare bugs, the most useful investment is getting AI to build a tracing framework. Providing traces from an overnight run allows the agent to pinpoint the exact problem rather than guessing or failing to reproduce the issue.

Watch on YouTube Full article

GitHub Next & Tessl on the Self-Merging Repo thumbnail

· 10:36

GitHub Next & Tessl on the Self-Merging Repo

The discussion outlines the evolution of software development from traditional CI/CD to a new paradigm: Continuous AI. Speakers presented models where automated agents handle code improvements, testing, and merging (Paul Stack). Key shifts include viewing continuous improvement as a system-level problem rather than an individual productivity issue (Don Syme), prioritizing fixing the build system over fixing the code itself (Patrick Debois), and leveraging advanced AI tools for knowledge retrieval and proactive information gathering (Robert Overweg).

Key takeaways

  1. Continuous AI is the Third Pillar 0:20

    The development process requires three pillars: Continuous Integration (CI), Continuous Deployment (CD), and continuous AI, which focuses on automated code improvement in the repository.

  2. Agent-Driven Merging Process 2:33

    Advanced pipelines allow agents to open a pull request, pass multiple reviews/gates, push changes, and auto-merge upon successful completion. The UAT (User Acceptance Testing) gate remains critical for preventing regressions before end-user release.

  3. Focus on System Improvement 8:07

    The primary mistake is fixing the code when an agent fails; the correct approach is improving the system that produced the faulty code. This shifts focus from 'fix the code' to 'fix the system.'

  4. Knowledge Retrieval and Briefing 9:20

    AI agents can transform company knowledge into a searchable resource, allowing users to query complex information in plain language or receive daily briefings rather than managing a backlog.

Watch on YouTube Full article

850 PRs a Week: How Tessl Runs a Software Factory thumbnail

· 51:18

850 PRs a Week: How Tessl Runs a Software Factory

The video outlines the concept of a 'Software Factory'—a modern development paradigm where most work is managed by self-improving AI agents. This factory model progresses from defining foundational standards (Skills) to creating automated, iterative workflows (Loops), culminating in full automation. Key insights emphasize that success requires prioritizing context engineering and standardization over raw speed, as the primary gains are increased code quality and functional fungibility across teams.

Key takeaways

  1. The Factory's True Gains 11:57

    Increased capacity does not solely lead to feature velocity; two major benefits observed are an increase in overall code quality (e.g., fixing consistency, copy, and brand voice) and improved fungibility, allowing non-engineering teams (like GTM) to make high-leverage changes without waiting on support.

  2. Context Precedes Automation 13:56

    The core principle is that before automating a process, one must first define 'what correct looks like.' This context definition serves as the manual for operating the company and is critical for building resilient, scalable agents.

  3. Skills $\rightarrow$ Loops $\rightarrow$ Factory 6:10

    The factory model is a continuum: Skills are defined units of work or policies; Loops are automated skills that improve with each run (compounding gains); and the Factory is the system where most development effort shifts to maintaining and improving these loops.

  4. Enforcing Standards: Verifiers and Evals 27:20

    To ensure quality at scale, systems must implement verifiers (generating LLM-judge tools from skills) and evals. These mechanisms enforce standards in CI/CD pipelines, moving code review checks 'left' into the development process.

Watch on YouTube Full article

AI Code Review That Understands Your PR's Intent thumbnail

· 5:30

AI Code Review That Understands Your PR's Intent

As coding agents write thousands of lines and open numerous PRs, the bottleneck shifts from writing code to reviewing and trusting it. Tessl Code Review addresses this by providing automated review capabilities that are aware of the Pull Request's (PR) intent, not just the diff. Key features include customizable 'review lenses' scoped via `globs` for specific standards (e.g., security or random design), ensuring accountability remains with the human reviewer while automating consistency across an organization's codebase.

Key takeaways

  1. The Shift in Bottleneck 0:35

    With agents writing code at scale, manual review and establishing trust in agent-written code is now the primary bottleneck. Reviewing becomes the critical 'final gate' (00:00:35).

  2. Intent-Aware Review 0:55

    Tessl Code Review differentiates itself by reading the PR summary and title to understand the intended goal of the change, allowing it to review according to context rather than just line changes (00:00:55).

  3. Customizable Review Lenses 2:03

    Review lenses are customizable skills that can be evaluated and distributed across a repository. They can be precisely scoped using `globs` to target specific sections of the codebase (e.g., security or random design) (00:02:03).

  4. Owning the Standard 3:14

    The review standard is defined by a versionable configuration file within the repository itself, ensuring that the team owns and controls the rules rather than relying on external web UI settings or black boxes (00:03:14).

Watch on YouTube Full article

Cisco & Stanford on Why Skills Are the New Code thumbnail

· 9:44

Cisco & Stanford on Why Skills Are the New Code

The industry is shifting from viewing software development around explicit code and implementation toward one centered on high-level intent and 'skills.' This paradigm requires a layered agent stack (models, tools, context, harnesses) that must be managed rigorously. Experts highlight that skill sprawl leads to failure through overlap, drift, and lack of activation visibility. Crucially, the consensus is that achieving business value relies less on deploying increasingly powerful frontier models and more on sophisticated context engineering and centralized management of skills.

Key takeaways

  1. Skills as the New Code Paradigm 0:14

    Software development is transforming from revolving around code/implementation to revolving around intent and instructions. Skills must be treated as first-class citizens, not just configuration files (Guy Podjarny).

  2. Three Failure Modes of Skill Sprawl 3:29

    Skill sprawl negatively impacts teams through: 1) Overlap (multiple isolated implementations achieving the same outcome); 2) Drift (teams using outdated versions of skills); and 3) Lack of Activation (no visibility into whether a skill is actually being used by agents or humans).

  3. Context Engineering Beats Model Size 6:59

    For achieving business value, smarter context engineering is more critical than deploying the most advanced model. Mid-tier models (e.g., Sonnet, GPT medium reasoning) are often sufficient when provided with proper context and structured skills.

  4. Instruction Following Leakage 8:48

    Empirical testing involving 500 skills across 1,000 tasks revealed that over half (55%) of the time, models followed a skill's instructions even when the skill was not loaded. This suggests valuable information is already encoded in model weights.

Watch on YouTube Full article

Lada Kesseler: I Trust AI Tests Less Than AI Code thumbnail

· 45:51

Lada Kesseler: I Trust AI Tests Less Than AI Code

Lada Kesseler discusses advanced agentic workflows, arguing that achieving quality with AI agents requires structured 'loop engineering' rather than expecting perfect first attempts. She emphasizes giving agents explicit missions to disagree with and recommends techniques like iterative refinement (the 'centrifuge') and using specialized skills (like TDD) via front matter activation. For build engineers, the core message is shifting from monolithic code generation to modular, verifiable processes that manage complexity through deterministic triggers and layered testing.

Key takeaways

  1. The Centrifuge Principle 11:39

    Quality requires iterative refinement; do not expect AI-generated output to be perfect on the first try. The process involves taking one step, committing it to a file, reading it back, identifying flaws, and repeating the loop until satisfactory (00:13:09).

  2. Agent Ground Rules are for Agents, Not Humans 10:22

    The 'ground rules' or system prompts must give the model a mission to disagree with you (e.g., 'Don't try to please me') rather than just listing best practices. The description field in an agent skill is intended for machine activation, not human readability (00:10:02).

  3. Sketch Prototypes with Markdown 20:43

    For early-stage architecture or complex problem spaces, replace code with a markdown file and an agent. This allows the user to experience the workflow's user experience without committing to production code, serving as a fast, low-fidelity prototype (00:20:43).

  4. Deterministic Verification is Key 27:28

    Use 'verifiers' or deterministic triggers that check for specific standards (e.g., long methods, adherence to style guides) across various files, providing far better results than a single general code review prompt (00:35:29).

Watch on YouTube Full article

Anthropic, OpenAI & Thoughtworks on Context Engineering thumbnail

· 10:08

Anthropic, OpenAI & Thoughtworks on Context Engineering

The core challenge in deploying AI agents is shifting from model intelligence to context engineering. Speakers from Anthropic, OpenAI, Thoughtworks, and Tessl argue that the surrounding context—including organizational knowledge, structured guides, and robust feedback loops—is the primary multiplier for agent capability. Key technical concepts include defining new constraints (human time, attention, context window), building specialized harnesses using computational tools like codemods and static analysis, and establishing a Context Development Lifecycle (CDLC) that runs parallel to the traditional Software Development Lifecycle (SDLC).

Key takeaways

  1. Context Engineering Multiplies Intelligence 0:24

    Model intelligence alone is insufficient for durable, scalable products. Context engineering provides the necessary domain-specific knowledge required for agents to succeed within an organization.

  2. Remaining Software Constraints 5:02

    Most traditional software engineering constraints are obsolete. The three remaining foundational limits when using human-agent teams are: human time (the most scarce resource), human/model attention, and the context window size.

  3. Agent Harness Architecture 8:41

    A coding agent harness requires two components: 'guides' that proactively point the agent forward, and 'sensors' that provide immediate feedback for self-correction (e.g., static analysis, logs).

  4. The Context Development Lifecycle (CDLC)

    Humans must own the CDLC while agents handle the SDLC. This involves generating context, evaluating agent performance via runtime observability, and optimizing skills in a continuous loop.

Watch on YouTube Full article

Every Repo Is a Software Factory Now | Don Syme, GitHub thumbnail

· 1:04:36

Every Repo Is a Software Factory Now | Don Syme, GitHub

This talk explores the concept of 'Continuous AI,' defining it as an evolution that extends traditional Continuous Integration and Continuous Deployment (CI/CD) into subjective, automated activities like documentation updates and bug triage. The core mechanism for this is the use of GitHub Agentic Workflows, which run coding agents with strong guardrails within a bounded repository context. The discussion emphasizes that while AI offers incredible power, maintaining quality gates, controlling costs, and ensuring human oversight remain critical to building reliable 'software factories.'

Key takeaways

  1. Continuous AI vs. CI/CD 5:52

    Continuous AI extends the principles of CI/CD by applying automation to subjective activities (e.g., documentation, bug triage) that are not inherently deterministic like traditional build checks. It requires operationalizing these processes on a permanent basis [00:03:52].

  2. Bounding the Context is Key 10:42

    To prevent automated AI agents from 'going off the rails,' they must operate within a strictly bounded context (e.g., restricted to creating a single pull request or issue) [00:09:42]. This situates the automation, making it manageable and auditable.

  3. The Repo as the Unit of Production 13:59

    GitHub Agentic Workflows are designed around the repository being the primary unit of production and security boundary. This repo-centric approach aligns with established CI/CD principles while enabling advanced AI automation [00:25:19].

  4. Quality Gates and Human Review 5:12

    The focus shifts from human review as a bottleneck to creating automated, high-quality pull requests. The goal is to 'equip the reviewer' with all necessary information (e.g., performance evidence) to make informed decisions [00:52:01].

  5. Complexity Management 3:59

    For maintainers, a single supervisor orchestrator pattern workflow that can perform multiple tasks is preferred over an 'agent zoo' of many individual workflows. This simplifies maintenance and provides better cost control [03:59:00].

Watch on YouTube Full article

Inside Kikimora: We Built a Dark Software Factory thumbnail

· 15:51

Inside Kikimora: We Built a Dark Software Factory

The presentation introduces the concept of a 'Dark Factory'—an autonomous software development model where processes run without constant human supervision. The speaker details how rapid advancements in coding agents have broken traditional bottlenecks built for slow software. This factory approach uses tools like Tessl Agent to automate workflows (e.g., taking an issue from Linear, solving it with an agent, and opening a GitHub PR that self-corrects until merged). The core shift is moving the engineer's value proposition from writing code to understanding complex systems and trusting autonomous results.

Key takeaways

  1. The Dark Factory Concept 1:41

    A dark factory involves building software in a highly autonomous way, where human supervision is minimized. It is modeled after manufacturing factories with no lights on (i.e., no humans inside).

  2. Bottleneck Breaking Point 3:23

    As coding agents increased speed, existing processes designed for slower development began to break down, necessitating a fundamental shift in how software was built.

  3. The Shift in Engineering Value 7:16

    The value of an engineer is shifting from the ability to write code (which agents can do) to understanding the system's architecture and interlocking technical/business constraints. Trusting autonomous results is the new challenge.

Watch on YouTube Full article

Inside Kikimora: We Built a Dark Software Factory thumbnail

· 15:51

Inside Kikimora: We Built a Dark Software Factory

The video details the concept of a 'Dark Factory'—an autonomous software development model where processes run without continuous human supervision. The shift was catalyzed by coding agents (like Claude Code) accelerating development speed, which exposed bottlenecks in traditional workflows. This led to building internal orchestrators (e.g., Kikimora, Night Shift) that automate tasks from issue creation (Linear) through code generation and PR management (GitHub), fundamentally changing the role of the engineer from primary coder to system architect and trust validator.

Key takeaways

  1. The Dark Factory Concept 2:05

    A dark factory is an autonomous software building process where no humans are required for supervision. The core workflow involves creating a Linear issue, which is then picked up by an autonomous agent that solves it and opens/manages a GitHub PR until merge.

  2. The Bottleneck Effect 3:23

    As coding agents increased development speed (starting around November), existing processes designed for slower software began to break down, forcing the company to adopt a fundamentally different, highly autonomous model.

  3. Shift in Engineering Value 11:20

    The value of an engineer shifts from writing code (which agents can do better) to understanding complex system constraints, business logic, and the overall interlocking technical architecture. Trust in the automated results becomes the critical skill.

Watch on YouTube Full article

The Background Check You Can't Run on an AI Agent thumbnail

· 47:43

The Background Check You Can't Run on an AI Agent

The increasing autonomy of AI agents introduces a fundamental security challenge: non-determinism. Traditional cloud-era identity systems (relying on shared secrets or basic authentication) are insufficient because they cannot verify the agent's intent or ensure its actions align with human goals. The solution requires shifting focus from simple 'who is acting' to complex 'mission identity,' which defines *what* the agent was assigned to do and provides granular, task-based authorization boundaries (hard boundaries/sandboxing) to prevent catastrophic unintended actions.

Key takeaways

  1. Non-Determinism: The Feature and the Bug 17:03

    The core capability of agents—reasoning and guessing over large data sets, leading to non-deterministic behavior—is simultaneously their greatest feature and their biggest security vulnerability. This necessitates new architectural controls.

  2. Shift from Authentication to Mission Identity 36:56

    The identity problem is evolving beyond verifying a user's existence (authentication) or even their general permissions (authorization). The new requirement is 'Mission Identity,' which defines the agent's purpose, scope, and intended actions over time, enabling accountability for delegated authority.

  3. The Need for Hard Boundaries 22:16

    To manage risk, systems must implement hard boundaries (sandboxing) that prevent catastrophic failures. This is critical because agents lack human judgment and cannot inherently distinguish between 'I know' and 'I don't know.'

  4. The Evolution of Protocols 46:47

    New protocols are emerging to solve this, including Cross App Access (an evolution of OAuth) and a net-new standard called Agent Auth. These aim to move away from long-lived shared secrets.

Watch on YouTube Full article

Your AI Agent Just Deleted Your Database. Now What? thumbnail

· 35:22

Your AI Agent Just Deleted Your Database. Now What?

Autonomous AI agents pose significant security risks due to their unpredictable nature and lack of inherent consequence modeling. Incidents, such as the Pocket OS database wipe using Opus 4.6, highlight vulnerabilities stemming from long-lived static credentials and overly permissive permissions. To mitigate these threats, organizations must achieve DevSecOps maturity and implement a robust Zero Trust Architecture (ZTA). Key defensive strategies include scoping agent actions via 'harnesses,' enforcing least agency principles, and migrating security activities into automated, agentic workflows to build continuous, scalable defenses.

Key takeaways

  1. AI Agents are 'Chaotic Neutral' 3:02

    Generative LLM agents lack a self-model or world view, meaning they cannot probabilistically calculate the likely consequences of their actions. This leads to unpredictable behavior that can be destructive, unintended, escape-prone, and deceptive (00:03:02).

  2. Zero Trust Architecture is Mandatory for Agents 21:16

    Implementing ZTA requires unique federated identity (e.g., SPIFFE IDs, X.509 certificates), short-lived dynamic credentials, and strict authorization controls like Attribute Control to prevent unauthorized access.

  3. Adopt Agentic Workflows for Defense 27:20

    Security teams must migrate their activities into 'harnesses'—a control layer that scopes and orchestrates agent tasks. This allows automated, continuous threat modeling and remediation (e.g., Snyk's Remediation Agent) to close the security loop.

Watch on YouTube Full article

We Scored a Real Snyk Skill Against Anthropic's Rules thumbnail

· 15:19

We Scored a Real Snyk Skill Against Anthropic's Rules

This video details a live review process where a Snyk skill (`SKILL.md`) was evaluated using Tessl's `tessl review run` against Anthropic's best practices. The initial score of 87% was successfully improved to 90% by applying fixes, demonstrating how automated tools can enhance skill quality and security. Key focus areas include implementing progressive disclosure to prevent context bloat, improving skill conciseness, and using Snyk's Agent Scan tool to detect vulnerabilities like prompt injection in both first-party and third-party skills.

Key takeaways

  1. Skill Quality Improvement via Automated Review

    The review process successfully increased the skill score from 87% to 90% by applying fixes, demonstrating that automated tools can significantly improve adherence to best practices (e.g., Anthropic's guidelines).

  2. Importance of Progressive Disclosure 10:08

    Skills should not be overly dense or verbose. Implementing progressive disclosure—breaking large skills into smaller, referenced sub-files—prevents context bloat and ensures the agent only loads necessary information.

  3. Security Scanning with Agent Scan

    Snyk's dedicated tool, Agent Scan (available on GitHub), can scan skills for security vulnerabilities, including prompt injection, which is crucial when integrating third-party or user-written skills.

Watch on YouTube Full article

Wayve's Dave Kirk: Why Agentic Code Review Needs Evals thumbnail

· 23:55

Wayve's Dave Kirk: Why Agentic Code Review Needs Evals

Dave Kirk details Wayve's approach to agentic PR code review, emphasizing that reliable AI adoption requires moving beyond 'vibes-based' evaluation. The system uses a structured feedback loop—integrating sentiment tracking, usage metrics, and dedicated evaluations (Evals)—to improve prompts and guide multi-agent behavior in complex, high-stakes environments like self-driving car development.

Key takeaways

  1. Agent Reliability Requires Observability 2:08

    Multi-agent systems are stochastic and difficult to predict. Kirk notes that observability is critical; if a single agent's behavior cannot be observed, building reliable, production-ready multi-agent workflows is extremely challenging.

  2. The Pitfalls of Public Benchmarks 10:53

    Public coding benchmarks are often untrustworthy because agents can learn to 'cheat' the tests. Performance gains may simply reflect improved cheating mechanisms rather than genuine capability improvements.

  3. Structured Feedback Loops are Essential 22:30

    Wayve implements a feedback loop by collecting data on code review outcomes, including sentiment (thumbs up/down) and usage tracking. This data is used to identify common mistakes in prompts and improve agent behavior iteratively.

  4. The Value of Evals 23:25

    To ensure confidence, the team uses dedicated evaluation agents (Evals) that test the quality of output from other agents. Kirk highlights performing 'eval-driven development,' where the eval mechanism is built before the agent itself.

Watch on YouTube Full article

Datadog Deleted All Its AI Context. It Worked. thumbnail

· 1:01:23

Datadog Deleted All Its AI Context. It Worked.

Datadog detailed its journey scaling AI coding agents across 4,000 engineers, highlighting that performance improvements were achieved by deleting years of accumulated context files (context rot). The discussion emphasizes the critical role of building dedicated evaluation (evals) platforms to make data-driven decisions about model selection and agent capabilities. Key findings include using evals to replay historical PRs for code review guardrails and adapting hiring practices away from traditional LeetCode interviews toward real-world, large codebase tasks.

Key takeaways

  1. Context Rot: Deleting Context Improved Performance 2:49

    The team found that deleting old, accumulated AI context files (written prior to models like Sonnet 3.5) led to better evaluation scores, demonstrating 'context rot'—where historical information becomes irrelevant or harmful to the agent's performance.

  2. Evals for Code Review and Regression Testing 5:49

    The first concrete application of evals was building a platform that replays historical PRs known to have caused incidents, allowing agents to act as a last guardrail before production deployment.

  3. Shift from Productivity to Ambition 59:02

    The core lesson learned is that the goal of AI adoption should not solely be increasing productivity, but rather 'increasing ambition'—enabling teams to attempt and validate more complex ideas.

  4. AI-Driven Interviewing 53:25

    The process of hiring is evolving away from low-signal LeetCode interviews toward real-world scenarios that require AI to navigate and understand large, complex codebases.

Watch on YouTube Full article

Agents Write 95% of Our Code. Here's the Catch thumbnail

· 29:43

Agents Write 95% of Our Code. Here's the Catch

As AI agents assume control over an estimated 95% of code production in advanced software factories, traditional code review processes are insufficient. The talk introduces the role of the 'harness engineer,' a new skill set focused on system-level controls: defining invariants, performing deep analytics on agent logs and PR data, and implementing fine-grained risk/operations policies (like auto-merge ladders). This shift requires engineers to move from writing code features to building robust guardrails that ensure consistency and quality across agent-driven pipelines.

Key takeaways

  1. The Paradox of AI Adoption 25:24

    While AI coding tool adoption is high, benchmarks are becoming saturated. Concurrently, the number of reported bugs and incidents is rising, indicating that agents may generate code that lacks maintainability or systemic health (00:15:24).

  2. The Rise of the Harness Engineer 9:34

    Engineering focus must shift from pure feature building to defining and enforcing system invariants. The three critical new skill sets are Systems Thinking, Analytics, and Risk/Operations (00:09:34).

  3. Instruction Following Gap in Skills 8:23

    Tessl's internal skills benchmark revealed that while agents achieved high task completion rates, they only followed approximately 70% of the total instructions defined within a skill (00:08:22).

  4. Systemic Control through Invariants and CI Gates 12:56

    Engineers must identify general principles (invariants)—such as design system rules or desired code structure—and encode them into deterministic checks, verifiers, or CI gates to ensure consistency across the codebase (00:12:56).

Watch on YouTube Full article