# AAIF Reading Group - Prompt Injection as Role Confusion: Rethinking Agent Security

## Executive summary

This reading group explores Prompt Injection through the lens of 'Role Confusion,' arguing that the vulnerability stems from the model's inability to strictly isolate roles (User, System, Tool, Thought). Speakers detail advanced attack vectors, including forging chain-of-thought reasoning and exploiting model style, which significantly increases attack success rates (ASR). Mitigation strategies discussed include implementing instruction hierarchies, sandboxing environments, and ensuring robust access controls, particularly when dealing with smaller or less state-of-the-art (SOTA) models.

## Key takeaways

- Role Confusion as the Core Vulnerability: The core claim is that prompt injection is a problem of role confusion, where the model treats fabricated reasoning (e.g., forged chain-of-thought) as its own internal thought process, rather than recognizing it as external input. This allows attackers to mislead the model into following harmful instructions.
- Advanced Attack Vectors: Attackers can increase ASR by injecting fabricated reasoning (forged thought) or by exploiting the model's style. One study showed that injecting forged chain-of-thought increased the ASR from 0% to 94% for OpenAI models.
- System-Level Security Risks: Security risks extend beyond prompts to system components. Examples include exploiting API calls (e.g., Anthropic's Opus 4.8) by injecting encrypted tokens/signatures from a smaller model (Haiku), and supply chain attacks (e.g., the LightLM incident) where malicious packages are introduced via open-source frameworks.
- Mitigation through Design: Proposed solutions include implementing 'instruction hierarchies' (prioritizing roles) and running models in isolated, sandboxed environments. The speakers emphasized that security must be baked into the design, not just added as a patch.

## Technical details

- Prompt Injection & Role Confusion: The vulnerability arises when the model fails to distinguish between roles (User, System, Tool, Thought), allowing malicious text to be interpreted as legitimate internal reasoning.
- Instruction Hierarchy: A mechanism for reducing prompt injection risk by placing roles in a defined hierarchy of priority. This approach was shown to reduce unsafe behavior detection from 60.6% to 0.07% in one paper.
- Role Probes: A technique involving training a linear classifier on content to measure how much a specific text segment (e.g., User vs. Tool) is related to the model's internal representation of that role.
- Supply Chain Attacks: Attackers compromise open-source frameworks (e.g., Trivy) to steal credentials and publish malicious packages (e.g., LightLM), allowing data exfiltration through seemingly legitimate CI/CD pipelines.

## Practical implications

- Security design must account for role confusion, treating it as a fundamental architectural flaw rather than just a prompt issue.
- When building agentic systems, implement sandboxing and strict access controls to limit the blast radius of a successful injection.
- Developers should be aware of advanced attacks, such as forging chain-of-thought reasoning, which can bypass standard guardrails.
- For production systems, consider implementing instruction hierarchies to enforce role priority and data integrity.

## Topics

Prompt Injection, Agent Security, LLM Vulnerabilities, Role Confusion, AI Architecture, Supply Chain Security, Prompt Injection as Role Confusion: Rethinking Agent Security, AAIF (Agentic AI Foundation)

Source: https://www.youtube.com/watch?v=ptejz5J4XhU