Topic

Prompt Injection

All digests tagged Prompt Injection

5 Best Practices for Building AI Agent Skills thumbnail

· 13:22

5 Best Practices for Building AI Agent Skills

This guide outlines five best practices for building reliable, secure, and effective AI agent skills. Skills are defined as procedural knowledge packaged in a `skill.md` file that teaches an AI agent specific job functions. Best practices emphasize improving skill triggering via detailed descriptions, grounding content in real domain expertise, managing context window size by using progressive disclosure, enforcing deterministic logic through scripts for critical steps, and rigorously vetting all skills for security vulnerabilities.

Key takeaways

  1. Best Practice 1: Optimize the Skill Description (Triggering) 2:19

    The agent uses the skill's name and description to decide if it should run. The description must be highly informative, stating what the skill does and when it should be used. It is recommended to 'oversell' the description slightly rather than underselling it, as models tend to under-trigger.

  2. Best Practice 2: Build from Real Expertise 5:58

    Skills must contain domain expertise that the model cannot generate on its own. This content should be synthesized from existing artifacts (e.g., old reports, run books, PR feedback). The highest value section in the skill body is often 'gotchas'—environment-specific facts or corrections made during manual execution.

  3. Best Practice 3: Spend Context Wisely 11:15

    The goal is to keep the skill body lean. Since the entire skill body contributes to the context window, only include information the agent wouldn't know otherwise. For large bodies of text, use a dedicated `references` sub-folder and implement 'progressive disclosure,' allowing the agent to open files only when needed.

  4. Best Practice 4: Use Deterministic Scripts for Fragile Steps

    For steps that must be exactly correct (fragile steps), do not rely on the model's probabilistic improvisation. Instead, write deterministic code and place it in a dedicated `scripts` directory within the skills folder. This ensures consistent, reliable execution.

  5. Best Practice 5: Vet Skills Before Running Them

    Treat agent skills like any external dependency package. Because skills can run code and access local file systems or APIs, they must be audited for security flaws (e.g., prompt injection or malware) before deployment.

Watch on YouTube Full article

AI Security in Practice: Protecting Your AI-Powered Applications - Olivia Liddell thumbnail

· 55:02

AI Security in Practice: Protecting Your AI-Powered Applications - Olivia Liddell

This talk provides a deep dive into securing AI-powered applications by addressing unique vulnerabilities that traditional security measures often miss. Using a fictional online retailer (ABC Company) as a case study, the presentation outlines three primary AI security risks—Prompt Injection, Data Poisoning, and Improper Output Handling. Mitigation requires implementing layered defenses across the entire application workflow: input validation, model protection, output sanitization, and continuous monitoring.

Key takeaways

  1. AI Attacks are Subtle and Invisible 17:32

    Unlike traditional attacks (e.g., DDoS), AI vulnerabilities can be slow, subtle, and partially invisible to standard firewalls or logging systems, requiring a shift in defensive thinking.

  2. Understand the Three Core Risks 17:32

    The three major risks are: Prompt Injection (tricking the model with disguised instructions), Data Poisoning (manipulating training data over time), and Improper Output Handling (trusting model output without validation).

  3. Implement Layered Defenses 45:00

    Security must be applied at four stages: Input Validation (sanitizing user input before it reaches the model), Model Protection (limiting scope and permissions), Output Sanitization (treating all AI-generated content as untrusted), and Monitoring/Operations (tracking model behavior changes).

Watch on YouTube Full article

Vibes Not Vulns: Securing the Era of AI-Written Software - Mackenzie Jackson thumbnail

· 23:21

Vibes Not Vulns: Securing the Era of AI-Written Software - Mackenzie Jackson

The integration of AI tools into software development introduces novel and complex security failure modes that traditional AppSec pipelines are unprepared for. The talk details how 'vibe coded' applications can ship insecure patterns, focusing heavily on prompt injection vulnerabilities within CI/CD workflows and the evolving risks in open-source supply chains (e.g., dependency hallucination). To mitigate these risks, guardrails must shift from simple code scanning to context-aware validation and strict access control.

Key takeaways

  1. AI Code Vulnerabilities 3:50

    AI systems are not perfect; they introduce vulnerabilities because they make assumptions about business logic. While models improve (especially with 'make sure it's secure' prompts), fundamental flaws like business logic errors remain, meaning AI code cannot be fully trusted yet.

  2. Prompt Injection in CI/CD 11:45

    A critical new vulnerability class is prompt injection, which allows an attacker to bypass system and application guardrails. This was demonstrated by exploiting the Gemini CLI tool within a CI/CD pipeline to achieve Remote Code Execution (RCE) and leak secrets from GitHub repositories.

  3. Supply Chain Risks 20:30

    Traditional vulnerability tracking using CVE numbers is fundamentally broken for modern malware attacks, which can spread rapidly. Furthermore, AI hallucination means package managers may suggest non-existent packages or outdated dependencies.

Watch on YouTube Full article

Prompt-Jacking: The Rise of a New Supply Chain Risk - Kasimir Schulz & Kenneth Yeung thumbnail

· 1:01:49

Prompt-Jacking: The Rise of a New Supply Chain Risk - Kasimir Schulz & Kenneth Yeung

The talk details how AI coding assistants and agentic systems introduce novel supply chain risks by allowing attackers to compromise developers through hidden text or malicious prompts. Vulnerabilities are often found not in traditional code flaws but in the design of agents themselves. Key attack vectors include indirect prompt injection via raw file data (e.g., READMEs), exploiting tool call responses, manipulating control sequences, and abusing structured formats like JSON within automated pipelines.

Key takeaways

  1. Inspect Raw Data Sources 2:00

    When reviewing codebases or documentation, always check the raw markdown/HTML data (e.g., a README file) rather than just the pre-rendered view, as malicious payloads can be hidden in comments or unrendered sections.

  2. Guardrails are Not Enough 10:30

    While guardrails (LLM as a judge, classification models) are useful, they can be bypassed by advanced techniques like recursive prompt injection or token manipulation. Security must address architectural weaknesses beyond just the prompt.

  3. Separate Control and Data Planes 17:45

    A critical defense is ensuring a strict separation between system instructions (the control plane) and user/tool input data (the data plane). LLMs often fail to distinguish between these two, making this separation crucial for security.

  4. Audit All Skills and Tools 28:00

    When using agentic skills or tools (e.g., in OpenWebUI), audit the skill's name, description, and parameters, as these are often inserted into the system prompt at a high privilege level and can be exploited for injection.

Watch on YouTube Full article

How to Break AI Systems (Before Someone Else Does) - Gary Lopez - NDC Toronto 2026 thumbnail

· 36:31

How to Break AI Systems (Before Someone Else Does) - Gary Lopez - NDC Toronto 2026

This talk provides an advanced overview of AI system vulnerabilities, arguing that Large Language Models (LLMs) are fundamentally different from traditional software because they predict the next token rather than executing code with clear separation. The presentation details various attack vectors—including direct and indirect prompt injections, encoding bypasses, and agentic risks—and outlines modern defense strategies like 'Spotlighting' to help developers build more resilient AI applications.

Key takeaways

  1. LLMs Fail Due to Data/Code Ambiguity 18:03

    Unlike traditional software with clear separation between executable code and data, LLMs treat all inputs (user prompts, system instructions, external data) as tokens for prediction. This fundamental flaw allows attackers to bypass safeguards.

  2. AI Red Teaming is Evolving 5:20

    Modern AI red teaming must consider not only traditional adversarial attacks but also Responsible AI (RAI) challenges, such as hallucinations, bias inference, and knowledge-to-action instructions.

  3. Mitigation Requires Multi-Layered Defense 29:50

    Defenses must be applied at multiple points: input filters (before the model receives data), output guards (after generation but before user display), and through techniques like 'Spotlighting' to enforce structural integrity.

Watch on YouTube Full article

BONUS EPISODE: 76 Malicious AI Skills Were Hiding in Plain Sight thumbnail

· 32:16

BONUS EPISODE: 76 Malicious AI Skills Were Hiding in Plain Sight

The security landscape for AI coding agents is rapidly evolving, shifting focus from traditional code vulnerabilities to risks introduced by 'Skills' and context management. The presentation details findings from Snyk’s ToxicSkills research, which uncovered 76 malicious skills containing malware or prompt injection vectors in open agent repositories. To mitigate these threats, organizations must adopt secure-by-default practices, implement governance through private skill registries (like Tessl), and utilize advanced scanning tools that analyze natural language context alongside code.

Key takeaways

  1. The Shift to Agent Security 6:38

    AI agents are changing developer workflows from reviewing lines of code to managing a 'team of agents.' This necessitates securing the inputs (context) and artifacts produced by these agents, making agent security paramount.

  2. Malicious Skills Threat 17:55

    Research found that third-party skills can contain purely malicious code or prompt injection attacks. These threats are difficult for traditional scanners to detect because they operate by influencing the agent's context rather than just being visible malware.

  3. Governance and Scanning Solutions 23:58

    Implementing secure skill registries (e.g., Tessl) that integrate security scanning (Snyk) is crucial. This allows organizations to scan skills for potential prompt injection, malware, and versioning issues before deployment.

  4. Secure-by-Default Stack 27:50

    Best practices include scanning third-party skills *before* use and running security scanners on custom-authored skills to identify potential vulnerabilities, such as mishandling or passing secrets in plaintext.

Watch on YouTube Full article

GPT-Red: Can AI red teams stop prompt injections? thumbnail

· 31:30

GPT-Red: Can AI red teams stop prompt injections?

This technical discussion explores how AI is being used in advanced cybersecurity defense mechanisms, specifically focusing on automated red teaming and scam interception. Key tools discussed include OpenAI's internal GPT-Red model, which significantly improves model resilience against prompt injections (e.g., reducing attack effectiveness from 95% to 10%). Another tool, ScamBuster, uses AI to bait scammers into revealing their tactics and infrastructure for threat intelligence gathering. The conversation concludes by addressing the widening gap between technical skill and raw ability in cybersecurity, warning that while AI provides immense power, human professionals must maintain foundational skills to remain effective.

Key takeaways

  1. GPT-Red's Effectiveness Against Prompt Injection 0:23

    OpenAI utilizes GPT-Red, an internal automated red teaming model, which performs better than human red teamers. This process was key in making models like GPT 5.6 Sol more robust; for instance, 'fake chain of thought attacks' that were 95% effective on GPT 5.1 are only 10% effective on GPT 5.6.

  2. ScamBuster for Threat Intelligence 5:21

    ScamBuster is an open-source AI tool designed to interact with email scammers, subtly gathering information about their tactics and infrastructure (IOCs) that can be fed back into security teams and law enforcement.

  3. The Skill vs. Ability Gap 10:35

    Bruce Schneier's essay highlights that AI is decoupling skills from abilities in cybersecurity, meaning individuals can now perform sophisticated hacks without the years of training and ethical framework traditionally required.

  4. Maintaining Foundational Skills 20:05

    The consensus takeaway for professionals is that while AI acts as a force multiplier, individuals must continue to develop their personal skills (e.g., the ability to blue/red team) to handle scenarios where the AI fails or cannot complete the task.

Watch on YouTube Full article