The Video Signal

Topic

Tessl

All digests tagged Tessl

· 14:35

We Scored Oracle's Database Skill Live: 95% Isn't Enough

The video demonstrates how Oracle's database team utilized AI 'skills'—encapsulated in a repository like `oracle/skills`—to guide AI agents in writing secure and best-practice compliant SQL. The process involved running an automated review (using Tessl) on a skill designed to improve database code generation, which initially scored 95%. Through iterative refinement and applying changes via a 'run, review, fix' cycle, the skill was successfully upgraded to a 100% success rate, showcasing a robust mechanism for encoding complex domain knowledge into AI workflows.

Key takeaways

  1. Purpose of Database Skills 5:51

    The primary function of the skill is to help agents write better SQL and database code by making them knowledgeable about the database's capabilities (e.g., property graphs, vector data). It helps discover what the database can do, rather than relying solely on general agent knowledge.

  2. Skill Review Process 13:44

    The skill was subjected to an automated review comparing it against best practices. The process involves running a 'run, review, fix' cycle, which applies changes and re-runs the validation in an agile style approach.

  3. Achieving 100% Compliance 14:35

    After making specific developer-suggested changes (e.g., adding a recommended sequence task), the skill's review score reached 100%, demonstrating continuous improvement and validation of domain knowledge.

Watch on YouTube Full article

· 58:39

Inside the Dark Factory: AI That Ships Code Solo

The video details Tessl's 'Dark Factory,' an autonomous system that handles a significant portion of their code shipping (65-70% of PRs). This factory uses AI agents and sophisticated verification layers to process tickets from inception (Linear) through to merge. The core architectural shift involves moving engineering focus from writing code to designing robust, automated workflows, emphasizing context management and layered verification to build trust in autonomous systems.

Key takeaways

  1. High Automation Rate 2:24

    Tessl routes 65-70% of its PRs through the Dark Factory. Historically, up to 95% of their codebase has never been reviewed by a human.

  2. Shift in Engineering Focus 17:12

    The role of the engineer shifts from implementing tickets to designing complex workflows, scoping tasks for agents, and encoding organizational 'taste' into verification layers (e.g., whiteboarding conversations).

  3. Trust is Earned, Not Enabled 31:00

    Building trust in the Dark Factory requires continuous effort, including running into failure modes and improving verification layers. Accountability remains with the person who filed the original ticket.

  4. Verification Layers are Key 22:22

    The most critical component is not the coding agent itself, but the layered verification system (Verifiers) that allow engineers to encode fuzzy principles (e.g., 'library should be the single source of truth') into deterministic checks.

Watch on YouTube Full article

· 53:34

Harness Engineering: Building an AI Software Factory

Harness engineering is a discipline focused on building automated loops of checks and maintenance agents that allow teams to delegate increasing amounts of codebase development to AI. The goal is not merely higher velocity, but achieving higher quality by shifting from manual code review (the primary bottleneck) to systematic process oversight. This involves tracking three key dimensions—autonomy, automation, and quality—and implementing layered validation systems: the Inner Loop (unit tests/linters), Outer Loop (agentic QA/UI testing), and Meta Loop (maintenance agents that analyze historical data for systemic improvements).

Key takeaways

  1. The Three Dimensions of Agent Adoption 10:39

    When adopting AI agents, track three metrics: Autonomy (how many human course corrections are needed); Automation (the level of oversight required, indicating trust); and Quality (ensuring the shipped product remains high quality). Progressing requires improving these dimensions sequentially.

  2. The Three Loops of Harness Engineering 19:04

    1. Inner Loop (Autonomy): Focuses on cheap, frequent checks like pedantic linting or unit tests to ensure agents get it right the first time. 2. Outer Loop (Automation): Involves slower, in-depth checks, such as agentic code review or running the product through a UI/CLI. 3. Meta Loop: Utilizes maintenance agents that analyze historical data (CI logs, PR comments) to propose systemic fixes and improvements to the entire process.

  3. The Primary Barrier is Organizational 28:20

    Harness engineering is fundamentally an organizational transformation, not just a technical one. Success requires changing workflows—for example, moving from monolithic PRs to smaller, low-risk chunks that can auto-merge, thereby shifting human behavior toward better practices.

Watch on YouTube Full article

· 32:16

BONUS EPISODE: 76 Malicious AI Skills Were Hiding in Plain Sight

The security landscape for AI coding agents is rapidly evolving, shifting focus from traditional code vulnerabilities to risks introduced by 'Skills' and context management. The presentation details findings from Snyk’s ToxicSkills research, which uncovered 76 malicious skills containing malware or prompt injection vectors in open agent repositories. To mitigate these threats, organizations must adopt secure-by-default practices, implement governance through private skill registries (like Tessl), and utilize advanced scanning tools that analyze natural language context alongside code.

Key takeaways

  1. The Shift to Agent Security 6:38

    AI agents are changing developer workflows from reviewing lines of code to managing a 'team of agents.' This necessitates securing the inputs (context) and artifacts produced by these agents, making agent security paramount.

  2. Malicious Skills Threat 17:55

    Research found that third-party skills can contain purely malicious code or prompt injection attacks. These threats are difficult for traditional scanners to detect because they operate by influencing the agent's context rather than just being visible malware.

  3. Governance and Scanning Solutions 23:58

    Implementing secure skill registries (e.g., Tessl) that integrate security scanning (Snyk) is crucial. This allows organizations to scan skills for potential prompt injection, malware, and versioning issues before deployment.

  4. Secure-by-Default Stack 27:50

    Best practices include scanning third-party skills *before* use and running security scanners on custom-authored skills to identify potential vulnerabilities, such as mishandling or passing secrets in plaintext.

Watch on YouTube Full article

· 16:56

Tessl Skills Clinic - Nnenna Ndukwe from Qodo

The session demonstrates how automated AI agent skills can be rigorously tested and improved using Tessl's review process against Anthropic's best practices. By applying human-guided refinements—particularly enhancing the skill description with natural trigger phrases and optimizing structure—the Qodo PR Resolver skill score jumped from 78% to 89%. This highlights that while AI generation is powerful, manual validation and refinement are crucial for achieving high-quality, reliable agent performance in production workflows.

Key takeaways

  1. Skill Description is Critical for Agent Triggering

    The skill's description metadata is identified as the single biggest factor determining if an agent will use it. Improving this by adding natural trigger phrases (e.g., 'coding issues,' 'code reviews') significantly increases discoverability and usage likelihood.

  2. Human Refinement Boosts Skill Quality

    Research suggests that while AI-generated skills are useful, validated and refined skills (human intervention) show a significant performance uplift. The average jump observed was approximately 14% for human-written skills.

  3. The Importance of Post-Fix Verification

    It is a best practice to include a validation step (e.g., running linters and static analysis) after an agent has applied fixes, ensuring the code remains compliant even after automated changes.

Watch on YouTube Full article

· 22:25

The DevOps Godfather on AI's "Dark Factory" Problem

Patrick Debois argues that the industry is approaching a 'dark factory' model of autonomous coding agents, mirroring historical skepticism around Continuous Delivery [0:02:51]. He emphasizes that success hinges not on better prompts or larger models, but on fundamentally restructuring teams and platforms. The developer role shifts from solo coder to agent orchestrator, requiring improvements in system-level engineering practices (e.g., building tooling for the agents) rather than just fixing the code produced by the agent [0:09:49]. Scaling this requires platform teams to centralize reusable components like skill registries and guardrails, moving organizations from a 'solo developer' model to a 'multiplayer system' [0:10:53].

Key takeaways

  1. Shift from Coder to Orchestrator 5:21

    The modern developer role evolves into that of an agent conductor or orchestrator, managing and guiding autonomous agents rather than writing all the code themselves. This requires a focus on system improvement over individual code fixes [0:02:51].

  2. Focus on Systemic Improvement 9:49

    The key mentality shift is to stop fixing the agent's output code and instead improve the underlying system, context, and tooling (harnesses/loops) that guide the agents [0:09:49].

  3. Platform Team Centralization 20:53

    To scale autonomous development, platform teams must own centralized components—such as skill registries, guardrails, and reusable harnesses—to prevent organizational sprawl and ensure consistency across teams [0:10:53].

  4. New Productivity Metrics

    Instead of measuring token spend, focus on two metrics: the number of human touches required for the agent to succeed, and the multiplier effect gained when a single system improvement benefits all users/teams [0:14:50].

Watch on YouTube Full article