Do Not Let Models Invent Numbers - Vinoo Ganesh, Kepler
Summary
Kepler addresses the critical challenge of using Large Language Models (LLMs) for mission-critical financial analysis, where numerical accuracy and auditability are paramount. The core architectural innovation is ensuring that models cite numbers but never write them. The system integrates deterministic code for calculations, a citation registry built from over 30 million SEC documents, and a post-processing filter that strips any figure that cannot be verifiably traced back to a source. This approach moves beyond standard Retrieval-Augmented Generation (RAG) to provide a guaranteed, auditable ground truth for financial data.
Key takeaways
-
Deterministic Architecture for Finance
The system architecture ensures that the LLM's role is limited to citing sources, while all mathematical operations are handled by deterministic, human-written code. A post-processing filter enforces this by stripping any unverified figures, making the platform structurally incapable of producing hallucinated numbers.
-
Structured Fact Annotation
13:47
To ensure traceability, every fact is annotated with metadata, including its value, unit, scale, and period. This allows the system to correctly contextualize numbers (e.g., identifying if a 'billion' refers to 2023 revenue or a 2025 filing).
-
Advanced Retrieval and Security
18:50
Kepler utilizes a hybrid search approach (combining BM25 and zero-entropy ANN embeddings) on TurboPuffer. Security is maintained through tenant isolation, where every document hit is reauthorized in Postgres to prevent data leakage across different client environments.
Technical details
-
Citation Registry and Data Ingestion
430s
The platform processes massive corpora of SEC data (over 30 million documents) to build a citation registry. This registry registers deterministic numerical facts, allowing users to click on any number and trace it back to its core source document.
-
Hybrid Search and Re-ranking
1237s
Information retrieval is achieved by combining BM25 and zero-entropy embeddings (1280 dimensions) within a namespace. The process involves reciprocal re-rankings and a cross-encoder to deconflict and prioritize the most trusted sources across multiple documents (e.g., 10-K vs. earnings call transcripts).
-
Model Limitations and Guarantees
The system provides a 'potent guarantee' by assigning unique identifiers to every fact. The platform refuses citations that cannot be verified, resulting in a high rate of rejected citations (e.g., 10,981 citations refused in internal evaluations), confirming that regular RAG platforms are insufficient for numerical accuracy.
Mentioned resources
- SEC filings (10K, 10Q)
- TurboPuffer
- Gemini Force
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.