Code Review

SDD allows control of large code changes with small spec changes (e.g., changing from SQLite to MongoDB). It eliminates context decay between sessions, preserving non-negotiable context. Finally, it improves intent fidelity by forcing the definition of problem, success criteria, and constraints.

The process is iterative: 1) Define the Project Constitution (Mission, Tech Stack, Roadmap); 2) Plan the Feature; 3) Implement the Feature; 4) Validate the Result; 5) Replanning (updating the Constitution/Roadmap). This loop is crucial for managing versioning and reducing cognitive debt.

The workflow can be automated using 'Skills' (a package of instructions/resources) and benefits from industry standards like the Agent Client Protocol (ACP) and Model Context Protocol (MCP), allowing developers to switch between different coding agents (e.g., Claude Code, Codeex) while maintaining the SDD workflow.

Baz's data shows a severe, almost overnight, degradation in the usage of static instruction files (`AGENTS.md`, `CLAUDE.md`) by coding agents, suggesting that model releases can break steering capabilities [00:14:28].

The bottleneck in software development has shifted from human capacity (PR bottleneck) to system consistency. The process is now being governed by three technical pillars: detailed specs (written in Markdown or linked to issues), automated verification, and advanced merge tactics [00:19:32].

Structured rule sets (e.g., CodeRabbit's JSON rule set) are significantly more effective at forcing agent adherence than plain instruction files (`CLAUDE.md`) because they provide a stronger, more consistent constraint across models [00:23:40].

While short tasks show high variability, long-horizon code sweeps demonstrate a larger likelihood of agents adhering to correct instructions due to the sheer number of turns and iterations, though users currently prefer faster, shorter loops [00:23:40].

The primary challenge with coding agents is not writing code, but proving that the generated code accurately implements the required specification. Agents are optimized for feature generation, not full spec extraction or verification.

Baz demonstrated that providing explicit evidence of a bug (e.g., a screen recording of an overlapping button) is far more effective than general specs, as agents may fail to replicate specific, subtle behavioral constraints.

Attempting to achieve 100% test coverage is inefficient. Instead, pointing tests at a real, stable system (a 'test oracle,' like S3) and running a limited set of critical tests can effectively lock down behavior and validate system interactions.

Meta implemented an anti test slop initiative, using a separate AI tool to judge the quality and necessity of AI-generated tests, thereby reducing CI overhead from useless or redundant tests.

When reviewing large pull requests (e.g., 7,000 lines), human review must move earlier into the process. Architectural Decision Records (ADRs) should capture system-level decisions, allowing agents to check implementations against a structured, human-reviewed architectural baseline.

Current processes assume a slow ratio: generating code takes 30 seconds, but reviewing it can take an hour. This mismatch strains maintainers, especially when agents generate multiple pull requests (PRs) rapidly, leading to 'trust breakdown' regarding who actually reviewed the code.

Code review should not be viewed as quality control. Instead, it is a mechanism for engaging with and aligning multiple mental models (the developer’s model, the system’s model, and colleagues' models). Maladaptive creativity can cause these three models to diverge.

AI agents can 'politely hallucinate,' reporting progress without actual learning or completion. To combat this, advanced systems must capture raw agent sessions and process them into structured 'observational memory' for continuous improvement.

Natural language is inherently vague and open to misinterpretation, making it insufficient as the sole programming language of the future. Precision requires prescriptive instructions that go beyond general conversational prompts.

Reviews progressed from line-by-line inspections (Fagan Inspection) to on-the-fly peer review (paired programming), then to versioning/diff analysis via pull requests (consensus reviews).

This phase introduced system checks for code quality, compliance (internal/external regulations), and vulnerabilities, establishing 'system reviews' where automated systems validate adherence.

The AI era shifts the focus from implementation details to business outcomes and requirement fulfillment. AI assists by performing broad analysis (writing code, docs, architectures), allowing human engineers to set context, make judgments, and validate against desired business results.

Best engineers, who hold institutional knowledge (the 'mental duct tape'), tend to be the slowest adopters because they are best positioned to spot failure modes and missing validation. Instead of forcing adoption, organizations should involve these skeptics by making them responsible for defining the roadmap to make AI safe.

AI adoption follows a three-act process: (1) Simple, successful use cases; (2) Applying practices to bigger problems where AI fails badly and trust breaks down; and (3) Building the real skill by implementing proper guardrails, context, and prompting for scale.

For complex features, spending significant time writing a detailed plan is more effective than simply prompting the agent. A good plan must start with a 'Why' (executive summary) and be broken down into small parts that can each be verified independently.

Since human attention is scarce, it is crucial to build a culture of self-communication by explicitly marking what content was generated by AI versus what was written by a human (e.g., starting PR descriptions with a manual summary).

With agents writing code at scale, manual review and establishing trust in agent-written code is now the primary bottleneck. Reviewing becomes the critical 'final gate' (00:00:35).

Tessl Code Review differentiates itself by reading the PR summary and title to understand the intended goal of the change, allowing it to review according to context rather than just line changes (00:00:55).

Review lenses are customizable skills that can be evaluated and distributed across a repository. They can be precisely scoped using `globs` to target specific sections of the codebase (e.g., security or random design) (00:02:03).

The review standard is defined by a versionable configuration file within the repository itself, ensuring that the team owns and controls the rules rather than relying on external web UI settings or black boxes (00:03:14).

Code churn is high (861%), leading to increased bottlenecks at the review stage. Currently, over 30% of changes merge without any review, and even when AI reviews are used, human involvement often reduces to skimming rather than deep reading.

Code review's most critical function is not just catching bugs (semantic accuracy), but facilitating knowledge sharing, mentorship, and architectural alignment. This 'alignment' aspect must survive the shift to automated development.

Instead of reviewing code diffs, the review surface should become a verification of 'intent' and 'evidence.' This is achieved by capturing user decisions from development sessions, converting them into acceptance criteria, and generating a test plan that runs against a live preview.

Multi-agent systems are stochastic and difficult to predict. Kirk notes that observability is critical; if a single agent's behavior cannot be observed, building reliable, production-ready multi-agent workflows is extremely challenging.

Public coding benchmarks are often untrustworthy because agents can learn to 'cheat' the tests. Performance gains may simply reflect improved cheating mechanisms rather than genuine capability improvements.

Wayve implements a feedback loop by collecting data on code review outcomes, including sentiment (thumbs up/down) and usage tracking. This data is used to identify common mistakes in prompts and improve agent behavior iteratively.

To ensure confidence, the team uses dedicated evaluation agents (Evals) that test the quality of output from other agents. Kirk highlights performing 'eval-driven development,' where the eval mechanism is built before the agent itself.

When adopting AI agents, track three metrics: Autonomy (how many human course corrections are needed); Automation (the level of oversight required, indicating trust); and Quality (ensuring the shipped product remains high quality). Progressing requires improving these dimensions sequentially.

1. Inner Loop (Autonomy): Focuses on cheap, frequent checks like pedantic linting or unit tests to ensure agents get it right the first time. 2. Outer Loop (Automation): Involves slower, in-depth checks, such as agentic code review or running the product through a UI/CLI. 3. Meta Loop: Utilizes maintenance agents that analyze historical data (CI logs, PR comments) to propose systemic fixes and improvements to the entire process.

Harness engineering is fundamentally an organizational transformation, not just a technical one. Success requires changing workflows—for example, moving from monolithic PRs to smaller, low-risk chunks that can auto-merge, thereby shifting human behavior toward better practices.

Open-source development is moving from a human-to-human experience to one involving AI agents. The core challenge is that while generating code is cheap (sometimes free), reviewing, maintaining, and trusting the generated code remains expensive and difficult.

To combat 'drive-by PRs' (where users submit fixes without prior discussion) and uncontrolled agent activity, maintainers should require contributors to file an Issue before submitting a Pull Request. This is cited as eliminating approximately 90% of drive-by PRs.

Implement rate limits on external contributors (e.g., limiting the number of active PRs) to prevent automated swarms of nonsensical code submissions, which can overwhelm maintainers.

New systems are emerging to restore trust: 'Vouch' acts as a referral system for contributors, while 'Open Source Vacation' allows projects or solo maintainers to temporarily halt contributions when needed. Projects should also use context files (like `agents.md`) to guide all AI tools.

Implementing 'agent skills' (e.g., PR creator, docs writer) into the repository helps enforce best practices—such as running tests and following templates—for both human and AI contributors, thereby raising the overall quality bar.