AI & Data Science Periodic Tables: How They Work Together
The video details the synergistic relationship between Data Science and Artificial Intelligence (AI), presenting both disciplines using 'Periodic Tables' as a conceptual framework. It emphasizes that modern AI applications are built upon robust data science foundations. A comprehensive example—Document Q&A—is used to illustrate a full pipeline, detailing how elements like Extract Transform Load (ET), Data Ingest (DI), and Data Cleansing (CD) prepare the data, which is then processed by AI components such as Embeddings (EM), Retrieval Augmented Generation (RAG), and Guardrails (GR). The process can be completed into a continuous loop using Drift Detection (DR) and Synthetic Data generation for continuous system improvement.
Key takeaways
-
AI relies on foundational data science work
0:25
The speaker notes that all advancements in AI sit atop the groundwork laid by data science, creating a feedback loop where models inform how data is prepared for future use. (0:15-0:30)
-
Data Science Pipeline Stages
1:38
The Data Science periodic table defines five groups across the top (Acquisition, Preparation, Modeling, Generation, Evaluation) and tracks data maturity through rows: Raw Data $\rightarrow$ Prepared Data $\rightarrow$ Model Data $\rightarrow$ Validated Insight. (1:30-2:25)
-
AI Pipeline Core Elements
2:40
The AI periodic table features groups like Retrieval and Orchestration, with core primitives including Prompt, Embed, and LLM. Key components include embeddings (encoding info into numbers) and RAG (coordinating retrieval). (2:35-3:40)
-
The Full Document Q&A Pipeline
3:30
Building a system requires combining elements from both tables. The process moves linearly through data preparation (ET $\rightarrow$ DI $\rightarrow$ CD $\rightarrow$ ST $\rightarrow$ EN $\rightarrow$ GO) and then AI processing (EM $\rightarrow$ Vx $\rightarrow$ RAG $\rightarrow$ PR $\rightarrow$ LG $\rightarrow$ GR). (4:30-6:20)
-
Closing the Loop for Continuous Improvement
6:20
To prevent the system from being a one-way street, the pipeline is closed using Data Drift (DR) detection and Synthetic Data generation. This allows the embedding model to fine-tune itself continuously based on failing patterns. (6:40-7:50)