# From Tokenmaxxing to Trusted Throughput — Mingsheng Hong, Ironclad

## Executive summary

The talk argues that optimizing AI token usage should not focus solely on cost reduction (austerity). Instead, the goal is to maximize 'Trusted Throughput'—the value derived from code validated by internal engineering and external customers. The speaker emphasizes that as AI makes code generation abundant, the bottleneck shifts downstream to Code Review and Continuous Integration (CI/CD). Key strategies include defining advanced metrics (e.g., weighted merged PRs) and improving developer experience by eliminating flaky tests and measuring wait times.

## Key takeaways

- AI Usage Dashboards as Smoke Detectors: Usage dashboards should track token usage across teams/individuals but must not be positioned as leaderboards or incentives for maximization. Instead, they serve as 'smoke detectors' to identify pockets of low adoption or sudden usage anomalies (1:34).
- Focus on Trusted Throughput, Not Cost Reduction: The metric for ROI should be 'Trusted Throughput'—high-quality output validated by internal engineering and external customers. Attempting to cut cost before measuring value is premature (8:33).
- Metrics Evolution Beyond Lines of Code (LOC): The process for measuring value has evolved from LOC, to open PRs, to merged PRs, and finally to weighted merged PRs that incorporate a complexity score. This moves the focus from volume to quality (11:20).

## Technical details

- Measuring Code Value and Complexity: The speaker details a metric evolution for code contribution: LOC $\rightarrow$ Open PRs $\rightarrow$ Merged PRs. To account for quality, merged PRs are now weighted by a complexity score (e.g., using an LLM to assign a T-shirt size) because a small concurrency fix can be more valuable than large boilerplate code (12:43).
- CI/CD Bottlenecks and Developer Experience: With abundant AI-generated code, the bottleneck shifts to Code Review and CI. Anti-patterns include submitting massive PRs due to slow regression tests. Solutions involve using AI tooling for initial review (style, missing tests) so human reviewers can focus on subjective judgment (architecture, security design). Key metrics include measuring the wait time from 'ready' to 'merged' and tracking flaky test occurrences (14:06; 16:56).
- Prompt Engineering Best Practices: To improve token efficiency, users should structure prompts by placing fixed system prompts at the top and varying content at the bottom. Context pruning is also critical to maintain efficient context management during long chat sessions (16:56).

## Practical implications

- Implement usage dashboards to track AI token cost and adoption gaps, but explicitly avoid using them for competitive ranking or leaderboards.
- Shift organizational focus from minimizing cost (austerity) to maximizing 'Trusted Throughput' by improving the quality of code review and validation processes.
- Invest in developer experience platforms to address CI bottlenecks, specifically by eliminating flaky tests and measuring the time elapsed between a PR being ready and it being merged.
- When building internal AI tools, prioritize defining an internal playbook (set of well-crafted prompts) for context-specific tasks rather than relying solely on external vendors.

## Topics

AI Token Optimization, CI/CD Best Practices, Software Metrics and Measurement, Developer Experience, Prompt Engineering, Ironclad

Source: https://www.youtube.com/watch?v=dSg0pu8d6qg