# We Vetted 2000 AI Skills Before They Reached Developers — Lucas Palma, Nubank

## Executive summary

Lucas Palma details how Nubank addressed AI skills—which function like supply chain dependencies—being a significant security risk in a regulated financial environment. To mitigate this, they built 'Skill Vector,' a comprehensive security review system that sits between the skill and the internal marketplace. This system vets every skill using a hybrid approach: deterministic checks (e.g., regex for unsafe instructions) followed by LLM analysis to assess context and behavior before granting permissions or allowing deployment.

## Key takeaways

- Treat AI Skills as Dependencies: AI skills must be treated like any other software dependency, requiring rigorous vetting and security gates, rather than just configuration files.
- Hybrid Security Scanning is Essential: The most effective approach combines deterministic checks (for easy risks) with LLM review (for contextual behavior analysis), ensuring comprehensive coverage.
- Security Gates Must Be Comprehensive: Vetting must cover the entire workflow, including local developer testing, CI/CD steps, PR feedback, and integration into a formal vulnerability management program.

## Technical details

- Skill Vector Architecture: Skill Vector is the security review tool implemented to check if an AI skill is safe for use. It operates as a CI step before skills are uploaded to the internal marketplace.
- Vetting Process Flow: The process involves: 1) Local developer iteration; 2) Pull Request (PR) submission; 3) Deterministic checks (using regex, etc.); 4) LLM review for context; 5) Generating a report in SARIF format; and 6) Final decision/approval gate.
- Security Risks Identified: Potential risks include hardcoded token requests leading to data leaks, instructions that prompt the use of dangerous shell commands, excessive permissions, and unintentional data exposure.

## Practical implications

- Implement mandatory security gates for all AI-generated or AI-integrated components (skills/plugins).
- Adopt a hybrid scanning approach combining static analysis (deterministic checks) with advanced contextual review (LLM).
- Enforce the practice of running skills locally before they are submitted to any shared marketplace.
- Integrate skill vetting into the existing vulnerability management program.

## Topics

AI Security, Supply Chain Risk Management, CI/CD, Software Engineering, Skill Vector, SARIF (Static Analysis Results Interchange Format), Nubank

Source: https://www.youtube.com/watch?v=iKQ78wyJEXU