# Doors of (AI)pportunity: The Front and Backdoors of LLMs - Kasimir Schulz & Kenneth Yeung

## Executive summary

The talk provides an advanced deep dive into LLM security vulnerabilities, moving beyond simple prompt injection attacks. Speakers demonstrate sophisticated methods including Knowledge Return Oriented Prompting (KROP), Policy Puppetry, and direct manipulation of the model's computational graph to implant backdoors. The discussion highlights that even seemingly safe formats (like ONNX) can be exploited if the underlying architecture is compromised, posing significant supply chain risks for AI-driven applications.

## Key takeaways

- Multiple Layers of Defense are Required: System prompts alone are insufficient protection. Developers must implement multiple layers of defense to mitigate attacks, as demonstrated by the failure of simple system prompt rules against advanced exploits.
- Indirect Prompt Injection is a Major Threat: Attacks do not require direct user interaction. Compromise can occur through injected text in external sources like customer reviews, emails (especially those processed by Retrieval Augmented Generation/RAG systems), or calendar invites.
- Model Backdoors are Difficult to Detect: Backdoors can be implanted into the computational graph of a model (e.g., changing an animal classification logic based on a specific color trigger). These backdoors can remain dormant until a precise trigger condition is met, making them highly effective and difficult to detect.

## Technical details

- Prompt Injection (Front Doors): This vulnerability occurs when trusted system prompts are combined with untrusted user input. Attackers can use phrases like 'ignore all your previous instructions' to override the model's intended behavior, leading to malicious output.
- Knowledge Return Oriented Prompting (KROP): KROP exploits implicit knowledge embedded in the LLM's training data (e.g., pop culture references like 'Little Bobby Tables'). By referencing this external knowledge, attackers can construct prompts that bypass explicit security rules and achieve malicious goals.
- Policy Puppetry: This is described as a universal and fully transferable alignment bypass. It works by exploiting the 'instruction hierarchy' (System > User > Document) using structured input, such as simulated XML tags, to completely override built-in safeguards or safety constraints.
- Computational Graph Backdoors: The most advanced attack involves manipulating the model's computational graph (the mathematical logic). By inserting conditional nodes, an attacker can force a specific output or behavior only when a precise trigger condition is met, bypassing all normal operational checks.
- Model Loading Vulnerabilities: When loading third-party models (e.g., from Hugging Face), developers must be wary of the `trust remote code` flag, as it allows arbitrary Python code execution during deserialization. Even seemingly safe formats like ONNX can be compromised if the underlying model structure is manipulated.

## Practical implications

- When integrating LLMs into CI/CD or agentic workflows, assume that all inputs (user prompts, external data sources, tool call results) are untrusted and potentially malicious.
- Implement robust input validation and sandboxing mechanisms to prevent prompt injection and ensure the model's context window cannot be hijacked by external text.
- For production models, conduct rigorous integrity checks on the computational graph structure and weights to detect subtle backdoors or unauthorized modifications.

## Topics

AI Security, Large Language Models (LLMs), Cybersecurity, Prompt Engineering, Machine Learning Architecture, Hugging Face, Netron.app

Source: https://www.youtube.com/watch?v=5XlD2gn8iIA