The Video Signal

Topic

Machine Learning

All digests tagged Machine Learning

· 19:50

Why Off-the-Shelf AI Doesn't Understand Money — Udi Menkes, Intuit

Off-the-shelf Large Language Models (LLMs) often provide fluent but inaccurate financial advice—a phenomenon termed the 'fluent bluff'—because they lack real-world experience. The core argument is that mere context is not sufficient; true intelligence requires grounding in verifiable business outcomes. To solve this, Intuit builds systems that transform raw business data (e.g., General Ledger, P&L) into millions of State-Action-Outcome records. These structured datasets are used to train models that predict the actual impact of actions, leading to 'outcome-driven finance' and outperforming frontier LLMs in real-world decision-making.

Key takeaways

  1. The Fluent Bluff: Why Off-the-Shelf Models Fail at Money 4:08

    LLMs trained on general internet data (blogs, books) can give confident advice that is fundamentally wrong for specific business constraints. Examples include recommending a negative cash flow business acquire another property or raising prices when it risks losing all revenue.

  2. The Solution: Grounding in State-Action-Outcome Data

    Instead of relying on general knowledge, the system must be grounded by creating structured records detailing what a company *did* (Action) given its current financial status (State), and what the verifiable result was (Outcome). This moves advice from theoretical possibility to proven reality.

  3. Measuring Impact: Beyond Simple Difference 15:45

    Determining the true impact of an action requires advanced statistical methods like Conditional Average Treatment Error (CATE) to account for confounding variables, such as a company's natural propensity for success. Failing to do this leads to inaccurate conclusions about causality.

Watch on YouTube Full article

· 10:52

Is Fine-Tuning Still Needed? LLMs, RAG, & LoRA

While early successes demonstrated that fine-tuning custom LLMs could outperform general models (e.g., legal AI in 2023), the landscape has shifted significantly. Modern frontier models are rapidly closing the gap due to massive context windows and improved reasoning capabilities. The current architectural best practice suggests prioritizing non-weight modification techniques like Retrieval Augmented Generation (RAG), Context Engineering, and Agent Skills before resorting to fine-tuning. Fine-tuning remains valuable for specific bottlenecks, such as achieving low latency or when using parameter-efficient methods like LoRA.

Key takeaways

  1. Fine-Tuning vs. General Models

    Historically, custom fine-tuned models outperformed off-the-shelf leaders (e.g., legal AI over GPT-4 in 2023). However, general models are catching up due to massive context windows and improved inference reasoning.

  2. Modern Customization Stack 6:20

    The preferred order for customization is: Base Model $\rightarrow$ Prompt/Context Engineering $\rightarrow$ RAG (for proprietary knowledge) $\rightarrow$ Agent Skills (for procedural know-how). Fine-tuning should be the last resort.

  3. Cost and Complexity of Customization 7:45

    Fine-tuning is costly, involving not only training runs but also data collection, evaluation, regression avoidance, and continuous maintenance as frontier models advance.

Watch on YouTube Full article

· 1:48:40

🔬 "The Most Innovative Diffusion Research Is Happening in Drug Discovery, Not Image Generation"

The video discusses the shift of advanced AI research from image generation into complex biological domains, specifically drug discovery and protein-ligand interaction modeling. Genesis Molecular AI introduces PEARL, a foundation model that uses diffusion techniques to predict how proteins flex to accommodate ligands (induced fit). The discussion highlights that modern drug design requires multi-parameter prediction (ADMET) and sophisticated agentic systems (SAPPHIRE) capable of reasoning like a chemist, moving far beyond simple structural predictions. Achieving high accuracy (sub-Angstrom resolution) is crucial for these models to be useful in physical chemistry workflows.

Key takeaways

  1. Diffusion Models in 3D Structure Prediction 1:42

    Diffusion techniques are proving to be a highly effective primitive for 3D structure prediction, particularly in modeling protein-ligand complexes. This represents a major advancement over previous methods like GANs and is central to Genesis's PEARL model.

  2. PEARL Model Capabilities 9:30

    The PEARL model predicts not only where a ligand binds but also models the conformational flexibility of the protein itself (induced fit). It demonstrated strong zero-shot performance on the OpenBind benchmark against notoriously difficult targets.

  3. Agentic Drug Discovery Systems 10:10

    Genesis's SAPPHIRE system represents an agentic approach to drug discovery. This AI agent is designed to mimic a chemist by reasoning about poses, forming hypotheses, reading literature, and proposing the next round of candidates.

  4. The Need for High Resolution 10:40

    Traditional benchmarks like 2Å RMSD are considered insufficient because they lack the resolution needed to discern critical details (e.g., aromatic ring flips) required for accurate physical chemistry predictions, necessitating a focus on sub-Angstrom accuracy.

Watch on YouTube Full article