Stop Prompting — Greg Pstrucha, Sentry
Summary
The talk argues that relying on manual re-prompting to correct AI-generated code is inefficient because agents lack memory and will repeat elementary mistakes. The core thesis is to stop prompting and start codifying rules and policies into deterministic checks. The speaker emphasizes that modern tooling, particularly linting, makes it cheap and easy to enforce complex conventions, such as keeping API endpoints synchronized with the OpenAPI schema, thereby raising the quality floor for AI-assisted development.
Key takeaways
-
Stop Prompting, Start Codifying
0:02
Instead of repeatedly correcting an agent's mistakes (e.g., overly defensive code, unnecessary endpoints), developers should codify the desired rules and policies into the codebase to ensure deterministic quality.
-
The Power of Linting
0:04
Linting is presented as the primary mechanism to codify rules. Historically, writing linters was costly, but with modern tools, it is cheap and easy, allowing developers to enforce conventions that prevent repeated agent errors.
-
Mining for Rules
0:05
A practical strategy is to mine past transcripts, GitHub PR reviews, and bot feedback to identify high-quality, deterministic rules that can be enforced via linting.
-
Handling Qualitative Measures
0:12
For rules that cannot be expressed deterministically (e.g., 'too many states' in a state machine), policies must be established. This can be managed using a 'Grumpy Engineer' review loop, which runs sub-agents against defined policies.
Technical details
-
API Schema Synchronization
6s
Lint rules can enforce that API endpoints (e.g., in Django/DRF) remain in sync with the OpenAPI schema. Specific rules include ensuring endpoints return a defined response type, requiring the OpenAPI schema decorator, and validating that the actual returned type matches the schema declaration.
-
Agent Skills and Linting
8s
When building agent skills that interact with APIs, linting can ensure that code blocks within the skill definition are type-checked, runnable in a sandbox, and adhere to the OpenAPI schema, preventing hallucinated or inaccurate examples.
-
Tooling and Implementation
10s
Recommended tools for implementing these checks include ESLint, ast-grep, and Taskless, which allows users to write the desired intent in English rather than writing the linter rule itself.
-
Warning on Metrics
14s
Developers must be wary of quantitative metrics like test coverage or cyclomatic complexity. Agents will optimize for the number (e.g., 100% coverage) even if the underlying tests are poor or lack value, creating a false sense of security.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.