# Why Off-the-Shelf AI Doesn't Understand Money — Udi Menkes, Intuit

## Executive summary

Off-the-shelf Large Language Models (LLMs) often provide fluent but inaccurate financial advice—a phenomenon termed the 'fluent bluff'—because they lack real-world experience. The core argument is that mere context is not sufficient; true intelligence requires grounding in verifiable business outcomes. To solve this, Intuit builds systems that transform raw business data (e.g., General Ledger, P&L) into millions of State-Action-Outcome records. These structured datasets are used to train models that predict the actual impact of actions, leading to 'outcome-driven finance' and outperforming frontier LLMs in real-world decision-making.

## Key takeaways

- The Fluent Bluff: Why Off-the-Shelf Models Fail at Money: LLMs trained on general internet data (blogs, books) can give confident advice that is fundamentally wrong for specific business constraints. Examples include recommending a negative cash flow business acquire another property or raising prices when it risks losing all revenue.
- The Solution: Grounding in State-Action-Outcome Data: Instead of relying on general knowledge, the system must be grounded by creating structured records detailing what a company *did* (Action) given its current financial status (State), and what the verifiable result was (Outcome). This moves advice from theoretical possibility to proven reality.
- Measuring Impact: Beyond Simple Difference: Determining the true impact of an action requires advanced statistical methods like Conditional Average Treatment Error (CATE) to account for confounding variables, such as a company's natural propensity for success. Failing to do this leads to inaccurate conclusions about causality.

## Technical details

- Data Modeling and Training: The system creates millions of vectors by combining financial data from sources like QuickBooks (General Ledger, P&L, Cash Flow statements) into State-Action-Outcome records. This structured data is used to train a Reinforcement Learning (RL) model that predicts optimal actions.
- Model Architecture: The process involves three steps: 1) Defining the business state; 2) Deriving potential actions from the state; and 3) Using an RL-trained model to predict outcomes, followed by training a final LLM layer to generate evidence-based advice.
- Performance Benchmarking: The grounded system was tested head-to-head against leading frontier models and outperformed them because of its access to verifiable outcome data, not just model size or general knowledge.

## Practical implications

- Architect AI systems to prioritize the capture and structuring of verifiable business outcomes (State-Action-Outcome) over simply feeding raw data into large models.
- Implement statistical methods like CATE when measuring the impact of interventions to accurately isolate causality from natural success factors.
- Focus on building proprietary 'systems of context'—unique, structured datasets derived from internal operations—as this represents the primary competitive moat in outcome-driven AI.

## Topics

AI, Finance Technology, Data Architecture, Machine Learning, Build Engineering, Princeton Study, QuickBooks, Turboax, Credit Karma, Mailchimp

Source: https://www.youtube.com/watch?v=Owb8g3yDyzo