# We let an AI agent execute Bash and lived to talk about it — Sarah Sanders, PostHog

## Executive summary

The talk details the architecture and security challenges of building agentic CLI tools, exemplified by PostHog's 'Wizard.' The speaker emphasizes that an agent with command execution capabilities is essentially a 'malware starter pack.' Security must be layered, deterministic, and must account for dangerous inputs originating not only from users but also from the system's own supply chain (the 'Context Mill'). The solution involves implementing a dedicated, deterministic scanner (Warlock) and ensuring that the LLM layer acts only as an 'adviser,' never an enforcer.

## Key takeaways

- The Danger of Agentic Tools: An agent that can run commands is structurally similar to malware, requiring extreme caution. The threat model must account for how 'attacks compose' from multiple, seemingly innocent vulnerabilities.
- Context Supply Chain Risk: The most dangerous input can come from the system's own supply chain (the Context Mill), such as a seemingly harmless prompt injection payload signed by the development team.
- Security Must Be Deterministic: For enforcement, security mechanisms must be deterministic and mechanical. The LLM layer should only be an 'adviser' to reduce noise, never the primary enforcer, to prevent failure modes.

## Technical details

- Agent Architecture: The Wizard is an agentic CLI that reads a codebase, installs SDKs, instruments events, and builds dashboards, reducing setup time from hours to minutes. It operates using models, prompts, and a set of tools.
- Security Scanner (Warlock): The Warlock is a standalone security scanner that runs on YARA rules, ensuring it is fully deterministic. It detects findings (e.g., exfiltration attempts) but does not act on them, keeping detection and enforcement separate.
- Security Layers: The security posture includes: 1) Deny-by-default rules; 2) An allow list (e.g., blocking random shell commands and reading .env files); 3) The Warlock scanner; 4) Triage (an LLM layer that advises on false positives); and 5) Telemetry.
- Prompt Injection Defense: To mitigate prompt injection, the speaker recommends matching not just a single word (like 'ignore'), but the verb plus an instruction-flavored noun in the rule's condition.

## Practical implications

- When building agents, assume the agent is a 'malware starter pack' and implement defense-in-depth.
- Always scan the entire content pipeline (source and use) for dangerous inputs, including content generated by your own supply chain.
- Separate detection (deterministic rules like YARA) from judgment (probabilistic LLM advice).
- When writing security rules, prioritize tracking real-world impact over how scary a vulnerability looks.

## Topics

AI Agents, Security Engineering, Prompt Injection, Supply Chain Security, Build Tools, Wizard, Warlock, YARA

Source: https://www.youtube.com/watch?v=4lXks428C9o