# Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket

## Executive summary

This talk details the engineering of a sophisticated Retrieval-Augmented Generation (RAG) assistant for a major Belgian financial client. The system processes a massive, complex corpus of over 270 PDFs (6,000 pages) containing rules, much of which is structured in tables. Key technical decisions included implementing a hybrid retrieval method (Dense Embeddings + BM25 via Reciprocal Rank Fusion, RRF) to achieve a 97% retrieval hit rate. The architecture utilizes a multi-tool agent to navigate the knowledge base, ensuring answers are highly accurate, fully cited, and presented in a user-friendly, streamed format.

## Key takeaways

- Handling Complex Source Material: The source corpus required specialized indexing: PDFs were chunked into ~2000-character segments, and tables were converted to CSV format to preserve the correct row/column relationships, which is crucial for accurate financial data extraction.
- Hybrid Retrieval for Accuracy: To improve retrieval accuracy, the system combined Dense Embeddings (for semantic meaning) with BM25 (for precise keywords/identifiers) using Reciprocal Rank Fusion (RRF). This boosted the retrieval hit rate from 92% to 97% on 63 questions.
- Agentic Workflow and Guardrails: The serving agent uses a multi-tool approach (Search -> Section -> Full Rulebook -> Directory Listing) with built-in guardrails. These limits prevent oversized rulebooks from flooding the context and prompt the agent to reconsider if a tool call adds new evidence.
- Optimizing User Experience (UX): To manage the median 38-second response time, the system streams Markdown text chunk-by-chunk. Structured components, like source cards (modeled using Pydantic), are streamed separately to avoid a large delay before the user sees any content.

## Technical details

- Indexing Pipeline: PDFs are chunked into segments (~2000 characters). Metadata is attached to each chunk, including the rulebook name, page number, and section path (discovered via regex, e.g., 'Chapter 3' or '4.2.1'). The process also involves generating LLM summaries for each chunk to identify the source document.
- Table Processing: Due to the high density of knowledge in tables, the system converts them into CSV format. This structured approach ensures the agent retrieves the correct row and column data, preventing misinterpretation of financial figures.
- Retrieval Architecture: The system employs a hybrid retrieval mechanism combining dense embeddings and BM25, fused using Reciprocal Rank Fusion (RRF). The agent is given four tools to widen the search scope, moving from specific search results to full sections or entire rulebooks.
- Evaluation and Testing: Evaluation was conducted using Promptfoo and human user testing. The speakers noted that LLM model evaluations can be noisy, and the standard template itself may sometimes be the source of the error, not the model.

## Practical implications

- When dealing with highly structured data (like financial rules), specialized preprocessing (e.g., converting tables to CSV) is necessary for accurate RAG retrieval.
- Implementing a multi-tool agent architecture with explicit guardrails (e.g., limiting context size) is crucial for maintaining reliability in complex knowledge domains.
- Streaming output (Markdown and structured components) is a critical UX pattern for maintaining user engagement during long-running AI tasks (e.g., 38-second median answer time).
- Evaluation pipelines must account for model noise and potential flaws in the evaluation template itself.

## Topics

Retrieval-Augmented Generation (RAG), Natural Language Processing (NLP), Information Retrieval, Agentic Systems, Build Engineering, Financial Technology (FinTech), Belgian Payments Rulebooks

Source: https://www.youtube.com/watch?v=gonAewBZxfU