# A Bank Trained Its Own Foundation Model to Catch Fraud — Freddy Lecue, Wells Fargo

## Executive summary

Wells Fargo implemented a proprietary Foundation Model using transformer architecture to process billions of customer transactions, moving beyond traditional language-based or specialized models like XGBoost. The model is pre-trained from scratch on raw, tabular transaction data (covering five dimensions: Who, What, How, When, Where). This single 'backbone' model can then be fine-tuned for diverse downstream tasks—including fraud detection, credit line decisions, and anti-money laundering—streamlining model development and improving performance metrics, such as reducing false positives while increasing detection rates, compared to the previous production XGBoost system.

## Key takeaways

- Foundation Model for Transactions: The model is designed not for language processing, but for structured, tabular data (transactions). It learns representations by predicting the next transaction/token, allowing a single backbone to be fine-tuned for multiple tasks (e.g., fraud detection, credit scoring).
- Outperforming Legacy Models: The new transformer-based model significantly improved fraud detection metrics, achieving a higher detection rate and a reduction in false positives compared to the existing production XGBoost model.
- Data Representation Challenge: The most complex engineering challenge was developing a robust tokenization strategy to convert heterogeneous tabular data (e.g., amounts, timing, zip codes, IP addresses) into a scalable vocabulary for the transformer model.

## Technical details

- Architecture and Training: The architecture uses transformer blocks (decoder-only) and is pre-trained from scratch (using zero weights) on the transaction data, rather than relying on existing open-weight models. The initial model size was 1 billion parameters, with plans for a next version at 11 billion parameters.
- Data Input and Feature Engineering: The model processes billions of transactions across five main dimensions: Who (customer/merchant), What (transacted item), How (channel: credit/debit/digital), When (timing), and Where (location/zip code). The goal was to identify core raw features rather than relying on pre-built, hard-crafted features.
- Tokenization Process: The process involves mapping quantitative, qualitative, and categorical data (like zip codes or transaction amounts) to a token and an ID, which is then passed through an embedding lookup. This is critical for scaling the vocabulary from highly varied sources.
- Inference and Latency: For downstream tasks like fraud classification, the transformer architecture captures the relationship between the current transaction and all past transactions (sequence). The system must maintain low latency, ideally under 250 milliseconds for real-time card approvals.

## Practical implications

- Reduces the need for building and maintaining multiple specialized models (e.g., separate models for fraud, AML, credit scoring).
- Enables the reuse of a single, robust foundation model across different business units within the financial institution.
- Provides significant cost savings and operational improvements by improving the balance between detection rate and false positive rate in fraud detection.
- The model's ability to process raw, structured data allows for deeper insights into customer behavior over time.

## Topics

Foundation Models, Transformer Architecture, Financial Technology (FinTech), Fraud Detection, Tabular Data Processing, Machine Learning Operations (MLOps), Freddy Lecue, Wells Fargo, AIE CODE summit

Source: https://www.youtube.com/watch?v=UB6hoWyh3B0