The Dark Arts of Skill Engineering — Paul Bakaus, Renaissance Geek
Summary
The talk, 'The Dark Arts of Skill Engineering,' argues that effective AI skill development moves beyond simple prompt engineering and into 'harness engineering.' The speaker details advanced techniques—such as using blind sub-agents, implementing anti-attractors, and utilizing mixture-of-experts architectures—to build robust, deterministic, and highly controllable skills. These techniques focus on exploiting the underlying capabilities of the AI model's execution environment (the 'harness') rather than just the input prompt, resulting in skills that are less susceptible to model drift and more reliable across different deployment environments.
Key takeaways
-
Harness Engineering vs. Prompting
0:06
Skills should be treated as extensions of the entire coding/AI harness, not just packaged prompts. This involves leveraging system capabilities like hooks, scripts, and standard output to enforce behavior, which is more reliable than prose instructions. (6:25)
-
Adversarial Sub-Agents
0:08
To achieve balanced critique or review, use two or more sub-agents that are blind to each other's output. This prevents a single model thread from anchoring on its own work or failing to detect obvious flaws. (7:55, 10:11)
-
Anti-Attractors for Divergence
0:17
To prevent model convergence on common or predictable outputs (the 'median'), implement anti-attractors. Techniques include shaving the next predicted token, generating random seeds (e.g., using celebrities for shaders), or running scripts that introduce unexpected input. (16:59)
-
Passive Guardrails via Hooks
0:20
Use hooks (e.g., pre-tool use hooks) to enforce design or code standards (like design linting) by actively preventing the model from writing non-compliant code, rather than relying on post-hoc corrections. (20:38)
-
Compounding Context and Memory
0:30
Skills can be made aware of prior sessions by saving intermediate results (e.g., critique files) in a dedicated folder. This allows subsequent runs to build context and track progress over time, enabling multi-session refactoring. (30:00)
Technical details
-
Sub-Agent Architecture
10s
The speaker demonstrated the use of two blind sub-agents (e.g., a 'Design Director' and a 'Deterministic Detector') synthesizing their findings into a balanced critique, which is superior to a single-threaded review. (10:11)
-
Scripting and Standard Output
32s
Instructions emitted on standard output (`stdout`) or via exit values are followed by models far more reliably than instructions buried in prose, though this technique can compromise prompt caching. (32:00)
-
Mixture of Experts (MoE) Skills
35s
Instead of one monolithic skill, complex tools should use an internal MoE model that routes based on the user's brief (e.g., switching between 'Product Design' and 'Brand Design' rules) and loads different underlying modules. (35:00)
-
Harness-Specific Overfitting
2s
Due to differences in model behavior (e.g., Gemini's love for animation, CodeX's love for rounded borders), skills must be compiled and tested for specific model/harness combinations (e.g., using dedicated XML blocks for Gemini vs. CodeX). (1:30, 4:00)
-
Evaluation and Testing (Evals)
45s
A robust skill requires an Evals harness that replicates the conditions of multiple target model harnesses (e.g., Claude Code, Gemini, CodeX). This includes running end-to-end tests, LLM-as-user simulations, and ablation testing (removing single lines of code to test impact). (45:00)
Mentioned resources
- Impeccable
- PBAus/impeccable-talks
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.