How to Break AI Systems (Before Someone Else Does) - Gary Lopez - NDC Toronto 2026
Summary
This talk provides an advanced overview of AI system vulnerabilities, arguing that Large Language Models (LLMs) are fundamentally different from traditional software because they predict the next token rather than executing code with clear separation. The presentation details various attack vectors—including direct and indirect prompt injections, encoding bypasses, and agentic risks—and outlines modern defense strategies like 'Spotlighting' to help developers build more resilient AI applications.
Key takeaways
-
LLMs Fail Due to Data/Code Ambiguity
18:03
Unlike traditional software with clear separation between executable code and data, LLMs treat all inputs (user prompts, system instructions, external data) as tokens for prediction. This fundamental flaw allows attackers to bypass safeguards.
-
AI Red Teaming is Evolving
5:20
Modern AI red teaming must consider not only traditional adversarial attacks but also Responsible AI (RAI) challenges, such as hallucinations, bias inference, and knowledge-to-action instructions.
-
Mitigation Requires Multi-Layered Defense
29:50
Defenses must be applied at multiple points: input filters (before the model receives data), output guards (after generation but before user display), and through techniques like 'Spotlighting' to enforce structural integrity.
Technical details
-
AI Vulnerability Taxonomy
430s
Risks include: Hallucinations (nonfactual information), Bias Inference, Privacy violations (PII extraction), Harmful Content Generation, Knowledge-to-Action Instructions (e.g., plotting an attack using Google Maps data), and Agentic Risks (multiple agents interacting).
-
Direct Prompt Injection Attacks
1230s
These attacks manipulate user control inputs to take over the model's actions or generated outputs. Vulnerabilities can exploit 'Competing Objectives' (e.g., starting a response with 'Sure, here is how to...') or 'Mismatch Generalization' by using encoded languages like Base64.
-
Indirect Prompt Injection Attacks
1430s
These attacks leverage external data sources (e.g., emails, PDFs, Reddit posts) that the model processes automatically. The payload can be imperceptible to humans (e.g., white text on a white background) but visible to the LLM/agent.
-
Spotlighting Techniques
1700s
A family of techniques designed to help models distinguish between data and code. Methods include: 1) Delimiters (e.g., using `<` and `>`), 2) Data Marking (adding special characters like ` ` or ` `), and 3) Encodings (applying Base64 encoding to user input).
Mentioned resources
- jailbroken: How does large language model safety training fail
- Crescendo Attack
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.