# The Dark Arts of Skill Engineering — Paul Bakaus, Renaissance Geek

## Executive summary

The talk, 'The Dark Arts of Skill Engineering,' argues that effective AI skill development moves beyond simple prompt engineering and into 'harness engineering.' The speaker details advanced techniques—such as using blind sub-agents, implementing anti-attractors, and utilizing mixture-of-experts architectures—to build robust, deterministic, and highly controllable skills. These techniques focus on exploiting the underlying capabilities of the AI model's execution environment (the 'harness') rather than just the input prompt, resulting in skills that are less susceptible to model drift and more reliable across different deployment environments.

## Key takeaways

- Harness Engineering vs. Prompting: Skills should be treated as extensions of the entire coding/AI harness, not just packaged prompts. This involves leveraging system capabilities like hooks, scripts, and standard output to enforce behavior, which is more reliable than prose instructions. (6:25)
- Adversarial Sub-Agents: To achieve balanced critique or review, use two or more sub-agents that are blind to each other's output. This prevents a single model thread from anchoring on its own work or failing to detect obvious flaws. (7:55, 10:11)
- Anti-Attractors for Divergence: To prevent model convergence on common or predictable outputs (the 'median'), implement anti-attractors. Techniques include shaving the next predicted token, generating random seeds (e.g., using celebrities for shaders), or running scripts that introduce unexpected input. (16:59)
- Passive Guardrails via Hooks: Use hooks (e.g., pre-tool use hooks) to enforce design or code standards (like design linting) by actively preventing the model from writing non-compliant code, rather than relying on post-hoc corrections. (20:38)
- Compounding Context and Memory: Skills can be made aware of prior sessions by saving intermediate results (e.g., critique files) in a dedicated folder. This allows subsequent runs to build context and track progress over time, enabling multi-session refactoring. (30:00)

## Technical details

- Sub-Agent Architecture: The speaker demonstrated the use of two blind sub-agents (e.g., a 'Design Director' and a 'Deterministic Detector') synthesizing their findings into a balanced critique, which is superior to a single-threaded review. (10:11)
- Scripting and Standard Output: Instructions emitted on standard output (`stdout`) or via exit values are followed by models far more reliably than instructions buried in prose, though this technique can compromise prompt caching. (32:00)
- Mixture of Experts (MoE) Skills: Instead of one monolithic skill, complex tools should use an internal MoE model that routes based on the user's brief (e.g., switching between 'Product Design' and 'Brand Design' rules) and loads different underlying modules. (35:00)
- Harness-Specific Overfitting: Due to differences in model behavior (e.g., Gemini's love for animation, CodeX's love for rounded borders), skills must be compiled and tested for specific model/harness combinations (e.g., using dedicated XML blocks for Gemini vs. CodeX). (1:30, 4:00)
- Evaluation and Testing (Evals): A robust skill requires an Evals harness that replicates the conditions of multiple target model harnesses (e.g., Claude Code, Gemini, CodeX). This includes running end-to-end tests, LLM-as-user simulations, and ablation testing (removing single lines of code to test impact). (45:00)

## Practical implications

- When designing complex AI skills, architect them using modular, multi-agent systems rather than relying solely on large system prompts.
- Implement explicit validation loops (hooks) to enforce constraints (e.g., design system rules, linting) at the point of code generation, making the process passive and continuous.
- For multi-user distribution, build a compiler or installer that can compile the skill to account for the unique behavioral differences and limitations of various target AI harnesses (e.g., CodeX, Claude Code, Gemini).
- Treat the AI skill as a system that must be rigorously tested across multiple model versions and environments using dedicated Evals harnesses.

## Topics

AI Engineering, Prompt Engineering, Harness Architecture, LLM Agents, Software Development Lifecycle (SDLC), Impeccable, PBAus/impeccable-talks

Source: https://www.youtube.com/watch?v=SQMCtZX3trg