# How to Break AI Systems (Before Someone Else Does) - Gary Lopez - NDC Toronto 2026

## Executive summary

This talk provides an advanced overview of AI system vulnerabilities, arguing that Large Language Models (LLMs) are fundamentally different from traditional software because they predict the next token rather than executing code with clear separation. The presentation details various attack vectors—including direct and indirect prompt injections, encoding bypasses, and agentic risks—and outlines modern defense strategies like 'Spotlighting' to help developers build more resilient AI applications.

## Key takeaways

- LLMs Fail Due to Data/Code Ambiguity: Unlike traditional software with clear separation between executable code and data, LLMs treat all inputs (user prompts, system instructions, external data) as tokens for prediction. This fundamental flaw allows attackers to bypass safeguards.
- AI Red Teaming is Evolving: Modern AI red teaming must consider not only traditional adversarial attacks but also Responsible AI (RAI) challenges, such as hallucinations, bias inference, and knowledge-to-action instructions.
- Mitigation Requires Multi-Layered Defense: Defenses must be applied at multiple points: input filters (before the model receives data), output guards (after generation but before user display), and through techniques like 'Spotlighting' to enforce structural integrity.

## Technical details

- AI Vulnerability Taxonomy: Risks include: Hallucinations (nonfactual information), Bias Inference, Privacy violations (PII extraction), Harmful Content Generation, Knowledge-to-Action Instructions (e.g., plotting an attack using Google Maps data), and Agentic Risks (multiple agents interacting).
- Direct Prompt Injection Attacks: These attacks manipulate user control inputs to take over the model's actions or generated outputs. Vulnerabilities can exploit 'Competing Objectives' (e.g., starting a response with 'Sure, here is how to...') or 'Mismatch Generalization' by using encoded languages like Base64.
- Indirect Prompt Injection Attacks: These attacks leverage external data sources (e.g., emails, PDFs, Reddit posts) that the model processes automatically. The payload can be imperceptible to humans (e.g., white text on a white background) but visible to the LLM/agent.
- Spotlighting Techniques: A family of techniques designed to help models distinguish between data and code. Methods include: 1) Delimiters (e.g., using `<` and `>`), 2) Data Marking (adding special characters like ` ` or ` `), and 3) Encodings (applying Base64 encoding to user input).

## Practical implications

- Implement robust input and output filters (e.g., toxic content detectors, prompt injection classifiers) at every stage of the AI pipeline.
- When building agentic applications, assume external data sources are untrusted vectors for attacks.
- Utilize techniques like Base64 encoding or structural delimiters to increase the difficulty of successful jailbreaks and prompt injections.

## Topics

AI Security, Red Teaming, Prompt Injection, Large Language Models (LLMs), Data Integrity, Responsible AI (RAI), jailbroken: How does large language model safety training fail, Crescendo Attack

Source: https://www.youtube.com/watch?v=9AG9lUdCgdY