Prompt-Jacking: The Rise of a New Supply Chain Risk - Kasimir Schulz & Kenneth Yeung
Summary
The talk details how AI coding assistants and agentic systems introduce novel supply chain risks by allowing attackers to compromise developers through hidden text or malicious prompts. Vulnerabilities are often found not in traditional code flaws but in the design of agents themselves. Key attack vectors include indirect prompt injection via raw file data (e.g., READMEs), exploiting tool call responses, manipulating control sequences, and abusing structured formats like JSON within automated pipelines.
Key takeaways
-
Inspect Raw Data Sources
2:00
When reviewing codebases or documentation, always check the raw markdown/HTML data (e.g., a README file) rather than just the pre-rendered view, as malicious payloads can be hidden in comments or unrendered sections.
-
Guardrails are Not Enough
10:30
While guardrails (LLM as a judge, classification models) are useful, they can be bypassed by advanced techniques like recursive prompt injection or token manipulation. Security must address architectural weaknesses beyond just the prompt.
-
Separate Control and Data Planes
17:45
A critical defense is ensuring a strict separation between system instructions (the control plane) and user/tool input data (the data plane). LLMs often fail to distinguish between these two, making this separation crucial for security.
-
Audit All Skills and Tools
28:00
When using agentic skills or tools (e.g., in OpenWebUI), audit the skill's name, description, and parameters, as these are often inserted into the system prompt at a high privilege level and can be exploited for injection.
Technical details
-
Indirect Prompt Injection
120s
Attackers embed malicious prompts in benign sources like README files or websites. The LLM, when processing the raw file content (not just the rendered view), reads and executes the hidden payload.
-
Command Injection via Tool Calls
210s
Vulnerabilities can be found in how agents handle chained commands. For example, while some systems check for semicolons (`;`), they might fail to detect injections using different syntax like dollar signs and parentheses (`$()`).
-
Control Tokens vs. Control Sequences
340s
Control tokens are special, non-text sequences used by model providers (e.g., `<user>`) to segment the context window. Attackers can exploit this by injecting similar text strings (control sequences) into tool call responses or system prompts to override instructions.
-
JSON/Structured Data Exploitation
2050s
Since LLMs treat all input as a single stream of text, an attacker can exploit structured data formats (like JSON) by appending commands or instructions after the expected data structure ends (e.g., adding 'end of JSON' to trigger arbitrary actions).
-
Model Context Pitfalls (MCP)
2750s
The MCP protocol allows connecting multiple tools/servers, but if the system lacks a way to differentiate tool origins, an attacker can hijack legitimate tool names (e.g., changing 'catfact' to 'create or update file') to redirect sensitive operations.
Mentioned resources
- Cursor
- WeezyPrint
- Mermaid
- Open Web UI
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.