Doors of (AI)pportunity: The Front and Backdoors of LLMs - Kasimir Schulz & Kenneth Yeung
Summary
The talk provides an advanced deep dive into LLM security vulnerabilities, moving beyond simple prompt injection attacks. Speakers demonstrate sophisticated methods including Knowledge Return Oriented Prompting (KROP), Policy Puppetry, and direct manipulation of the model's computational graph to implant backdoors. The discussion highlights that even seemingly safe formats (like ONNX) can be exploited if the underlying architecture is compromised, posing significant supply chain risks for AI-driven applications.
Key takeaways
-
Multiple Layers of Defense are Required
35:35
System prompts alone are insufficient protection. Developers must implement multiple layers of defense to mitigate attacks, as demonstrated by the failure of simple system prompt rules against advanced exploits.
-
Indirect Prompt Injection is a Major Threat
43:25
Attacks do not require direct user interaction. Compromise can occur through injected text in external sources like customer reviews, emails (especially those processed by Retrieval Augmented Generation/RAG systems), or calendar invites.
-
Model Backdoors are Difficult to Detect
50:15
Backdoors can be implanted into the computational graph of a model (e.g., changing an animal classification logic based on a specific color trigger). These backdoors can remain dormant until a precise trigger condition is met, making them highly effective and difficult to detect.
Technical details
-
Prompt Injection (Front Doors)
1035s
This vulnerability occurs when trusted system prompts are combined with untrusted user input. Attackers can use phrases like 'ignore all your previous instructions' to override the model's intended behavior, leading to malicious output.
-
Knowledge Return Oriented Prompting (KROP)
1790s
KROP exploits implicit knowledge embedded in the LLM's training data (e.g., pop culture references like 'Little Bobby Tables'). By referencing this external knowledge, attackers can construct prompts that bypass explicit security rules and achieve malicious goals.
-
Policy Puppetry
2035s
This is described as a universal and fully transferable alignment bypass. It works by exploiting the 'instruction hierarchy' (System > User > Document) using structured input, such as simulated XML tags, to completely override built-in safeguards or safety constraints.
-
Computational Graph Backdoors
2805s
The most advanced attack involves manipulating the model's computational graph (the mathematical logic). By inserting conditional nodes, an attacker can force a specific output or behavior only when a precise trigger condition is met, bypassing all normal operational checks.
-
Model Loading Vulnerabilities
2405s
When loading third-party models (e.g., from Hugging Face), developers must be wary of the `trust remote code` flag, as it allows arbitrary Python code execution during deserialization. Even seemingly safe formats like ONNX can be compromised if the underlying model structure is manipulated.
Mentioned resources
- Hugging Face
- Netron.app
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.