A Bank Trained Its Own Foundation Model to Catch Fraud — Freddy Lecue, Wells Fargo
Summary
Wells Fargo implemented a proprietary Foundation Model using transformer architecture to process billions of customer transactions, moving beyond traditional language-based or specialized models like XGBoost. The model is pre-trained from scratch on raw, tabular transaction data (covering five dimensions: Who, What, How, When, Where). This single 'backbone' model can then be fine-tuned for diverse downstream tasks—including fraud detection, credit line decisions, and anti-money laundering—streamlining model development and improving performance metrics, such as reducing false positives while increasing detection rates, compared to the previous production XGBoost system.
Key takeaways
-
Foundation Model for Transactions
The model is designed not for language processing, but for structured, tabular data (transactions). It learns representations by predicting the next transaction/token, allowing a single backbone to be fine-tuned for multiple tasks (e.g., fraud detection, credit scoring).
-
Outperforming Legacy Models
The new transformer-based model significantly improved fraud detection metrics, achieving a higher detection rate and a reduction in false positives compared to the existing production XGBoost model.
-
Data Representation Challenge
12:39
The most complex engineering challenge was developing a robust tokenization strategy to convert heterogeneous tabular data (e.g., amounts, timing, zip codes, IP addresses) into a scalable vocabulary for the transformer model.
Technical details
-
Architecture and Training
445s
The architecture uses transformer blocks (decoder-only) and is pre-trained from scratch (using zero weights) on the transaction data, rather than relying on existing open-weight models. The initial model size was 1 billion parameters, with plans for a next version at 11 billion parameters.
-
Data Input and Feature Engineering
558s
The model processes billions of transactions across five main dimensions: Who (customer/merchant), What (transacted item), How (channel: credit/debit/digital), When (timing), and Where (location/zip code). The goal was to identify core raw features rather than relying on pre-built, hard-crafted features.
-
Tokenization Process
759s
The process involves mapping quantitative, qualitative, and categorical data (like zip codes or transaction amounts) to a token and an ID, which is then passed through an embedding lookup. This is critical for scaling the vocabulary from highly varied sources.
-
Inference and Latency
For downstream tasks like fraud classification, the transformer architecture captures the relationship between the current transaction and all past transactions (sequence). The system must maintain low latency, ideally under 250 milliseconds for real-time card approvals.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.