The Video Signal

Topic

Pipeline Engineering

All digests tagged Pipeline Engineering

Processing Documents: Jev vs OSS Models thumbnail

· 16:48

Processing Documents: Jev vs OSS Models

This video compares several model architectures (Jev, Quen, Leia, Jeff) for automating decision-making and classification tasks within document processing pipelines. The core finding is that the open decoder model, Quen, performs remarkably close to the generalized classifier, Jev, across various tasks (language detection, document classification, document routing). The approach demonstrates that robust classification can be achieved without requiring pre-training on specific labels, making it highly valuable for building flexible, automated document pipelines.

Key takeaways

  1. Quen's Performance in Classification

    The Quen decoder model, which is a standard Language Model (LM) and not specifically trained for classification, showed accuracy very close to Jev in tasks like language detection and document routing, suggesting its utility for general document understanding.

  2. Model Architecture Comparison 7:16

    Jev is a generalized classifier trained with reinforcement learning, optimized for confidence scoring. Quen is a decoder LM. Leia and Jeff are encoder models (Leia predicts a mask token; Jeff is good for entity/relation extraction). While encoders are smaller and easier to fine-tune, they were less accurate than the decoders in this demonstration.

  3. Document Processing Tasks Covered

    The models were tested on five key document pipeline tasks: Language Detection (using Lingua as baseline), Orientation Detection (using Tesseract as baseline), Document Classification, Document Splitting, and Document Routing/Triage.

Watch on YouTube Full article

Build A Reasoning Model Scratch 3: The Verifier for Evaluation and RL with Verifiable Rewards thumbnail

· 1:26:47

Build A Reasoning Model Scratch 3: The Verifier for Evaluation and RL with Verifiable Rewards

This video details the implementation of a robust verifier pipeline for evaluating Large Language Models (LLMs) on mathematical benchmarks, specifically using the MATH-500 dataset. The verifier is crucial for establishing a baseline performance metric, which will later be used in Reinforcement Learning with Verifiable Rewards (RLVR) training. The process involves eight key steps: generating text, extracting the final answer (ideally from a boxed format), normalizing the answer to a canonical form, mathematically verifying its equivalence to a ground truth using `sympy`, and finally grading the answer to compute overall model accuracy.

Key takeaways

  1. LLM Evaluation Methodologies 2:01

    Model evaluation can be categorized into Multiple Choice, Verifier-based, Leaderboard-based, and LLM Judge methods. The verifier approach is preferred here because it provides an objective evaluation with a hard ground truth answer, which is necessary for RLVR training.

  2. The Verifier Pipeline 19:13

    The evaluation pipeline is complex, requiring steps to extract the final answer, normalize the format (e.g., removing LaTeX fluff), verify mathematical equivalence, and grade the result. This robustness is critical for reliable benchmarking.

  3. Model Performance Comparison 27:13

    The base model's accuracy (e.g., 15.6% on MATH-500) is significantly lower than the reasoning model's accuracy (e.g., 50.8%). This highlights the value of specialized reasoning techniques and training.

  4. Reproducibility Caveats 20:15

    Model evaluation results can vary based on the computing device (CPU, MPS, CUDA) and due to floating-point math, necessitating running evaluations multiple times and averaging the results for robustness.

Watch on YouTube Full article

AI & Data Science Periodic Tables: How They Work Together thumbnail

· 13:21

AI & Data Science Periodic Tables: How They Work Together

The video details the synergistic relationship between Data Science and Artificial Intelligence (AI), presenting both disciplines using 'Periodic Tables' as a conceptual framework. It emphasizes that modern AI applications are built upon robust data science foundations. A comprehensive example—Document Q&A—is used to illustrate a full pipeline, detailing how elements like Extract Transform Load (ET), Data Ingest (DI), and Data Cleansing (CD) prepare the data, which is then processed by AI components such as Embeddings (EM), Retrieval Augmented Generation (RAG), and Guardrails (GR). The process can be completed into a continuous loop using Drift Detection (DR) and Synthetic Data generation for continuous system improvement.

Key takeaways

  1. AI relies on foundational data science work 0:25

    The speaker notes that all advancements in AI sit atop the groundwork laid by data science, creating a feedback loop where models inform how data is prepared for future use. (0:15-0:30)

  2. Data Science Pipeline Stages 1:38

    The Data Science periodic table defines five groups across the top (Acquisition, Preparation, Modeling, Generation, Evaluation) and tracks data maturity through rows: Raw Data $\rightarrow$ Prepared Data $\rightarrow$ Model Data $\rightarrow$ Validated Insight. (1:30-2:25)

  3. AI Pipeline Core Elements 2:40

    The AI periodic table features groups like Retrieval and Orchestration, with core primitives including Prompt, Embed, and LLM. Key components include embeddings (encoding info into numbers) and RAG (coordinating retrieval). (2:35-3:40)

  4. The Full Document Q&A Pipeline 3:30

    Building a system requires combining elements from both tables. The process moves linearly through data preparation (ET $\rightarrow$ DI $\rightarrow$ CD $\rightarrow$ ST $\rightarrow$ EN $\rightarrow$ GO) and then AI processing (EM $\rightarrow$ Vx $\rightarrow$ RAG $\rightarrow$ PR $\rightarrow$ LG $\rightarrow$ GR). (4:30-6:20)

  5. Closing the Loop for Continuous Improvement 6:20

    To prevent the system from being a one-way street, the pipeline is closed using Data Drift (DR) detection and Synthetic Data generation. This allows the embedding model to fine-tune itself continuously based on failing patterns. (6:40-7:50)

Watch on YouTube Full article