# Prompt-Jacking: The Rise of a New Supply Chain Risk - Kasimir Schulz & Kenneth Yeung

## Executive summary

The talk details how AI coding assistants and agentic systems introduce novel supply chain risks by allowing attackers to compromise developers through hidden text or malicious prompts. Vulnerabilities are often found not in traditional code flaws but in the design of agents themselves. Key attack vectors include indirect prompt injection via raw file data (e.g., READMEs), exploiting tool call responses, manipulating control sequences, and abusing structured formats like JSON within automated pipelines.

## Key takeaways

- Inspect Raw Data Sources: When reviewing codebases or documentation, always check the raw markdown/HTML data (e.g., a README file) rather than just the pre-rendered view, as malicious payloads can be hidden in comments or unrendered sections.
- Guardrails are Not Enough: While guardrails (LLM as a judge, classification models) are useful, they can be bypassed by advanced techniques like recursive prompt injection or token manipulation. Security must address architectural weaknesses beyond just the prompt.
- Separate Control and Data Planes: A critical defense is ensuring a strict separation between system instructions (the control plane) and user/tool input data (the data plane). LLMs often fail to distinguish between these two, making this separation crucial for security.
- Audit All Skills and Tools: When using agentic skills or tools (e.g., in OpenWebUI), audit the skill's name, description, and parameters, as these are often inserted into the system prompt at a high privilege level and can be exploited for injection.

## Technical details

- Indirect Prompt Injection: Attackers embed malicious prompts in benign sources like README files or websites. The LLM, when processing the raw file content (not just the rendered view), reads and executes the hidden payload.
- Command Injection via Tool Calls: Vulnerabilities can be found in how agents handle chained commands. For example, while some systems check for semicolons (`;`), they might fail to detect injections using different syntax like dollar signs and parentheses (`$()`).
- Control Tokens vs. Control Sequences: Control tokens are special, non-text sequences used by model providers (e.g., `<user>`) to segment the context window. Attackers can exploit this by injecting similar text strings (control sequences) into tool call responses or system prompts to override instructions.
- JSON/Structured Data Exploitation: Since LLMs treat all input as a single stream of text, an attacker can exploit structured data formats (like JSON) by appending commands or instructions after the expected data structure ends (e.g., adding 'end of JSON' to trigger arbitrary actions).
- Model Context Pitfalls (MCP): The MCP protocol allows connecting multiple tools/servers, but if the system lacks a way to differentiate tool origins, an attacker can hijack legitimate tool names (e.g., changing 'catfact' to 'create or update file') to redirect sensitive operations.

## Practical implications

- Implement robust input sanitization and validation that specifically checks for control sequences or structured data format breaks.
- Prioritize the separation of the control plane (system instructions) from the data plane (user/tool inputs) in all agentic systems.
- When building agents, ensure that tool responses are processed by a dedicated guardrail layer that cannot be bypassed by simple text injection.
- For critical applications, consider caching system prompts to reduce token costs while maintaining security separation.

## Topics

Prompt Injection, Agentic AI Security, Supply Chain Risk, Control Tokens/Sequences, LLM Guardrails, Data Exfiltration, Cursor, WeezyPrint, Mermaid, Open Web UI

Source: https://www.youtube.com/watch?v=SsTtp_Nrmn4