The Video Signal

Topic

Data Engineering

All digests tagged Data Engineering

Ask the Experts: Inside Nemotron Post-Training | Nemotron Labs thumbnail

· 51:07

Ask the Experts: Inside Nemotron Post-Training | Nemotron Labs

This session details the advanced post-training pipeline used by NVIDIA AI researchers to build state-of-the-art (SOTA) models like Nemotron, focusing on enhancing intelligence and enabling agentic capabilities. Post-training is presented as an evolution from traditional task-specific training, utilizing structured data, chat templates, and specialized frameworks like NeMo Gym and NeMo RL. Key strategies discussed include performing ablation studies on data subsets, employing Mixture of Experts (MoE) for capability generalization, and establishing robust feedback loops using real-world user data to prevent model degradation.

Key takeaways

  1. Post-Training Evolution 3:58

    Post-training builds upon pre-training (which uses massive, diverse, unstructured data for causal language modeling) by focusing on structured data. This teaches the model to follow instructions, use chat templates, and emit tool calls, moving beyond simple task-specific or multi-task training.

  2. Structured Data for Capabilities 7:29

    To narrow model focus, post-training emphasizes structured data (e.g., tool responses, user prompts, tool sets) over unstructured text. This allows the model to learn specific formats, such as JSON or XML, for tool interaction.

  3. Mitigating Capability Degradation 13:54

    To specialize a model (e.g., for software engineering) without losing general capabilities, techniques include including general chat data in the blend and using algorithmic approaches like multi-teacher on-policy distillation (MOPD).

  4. Data Bias and Profiling 19:20

    Data quality is paramount. Researchers must analyze data statistics (e.g., trajectory length, tool call diversity) and perform ablation studies to identify and address biases (e.g., over-reliance on a single tool) that could cause model failure in real-world use.

  5. Starting the Pipeline 25:12

    For developers, the process involves defining the model's 'soul' (core capabilities), quantifying these goals via benchmarks, setting quantitative milestones, and iterating through the post-training pipeline. Starting resources include Nemotron 3.5 Lightning.

Watch on YouTube Full article

How do you diffuse AI into the real world? — Varun Shenoy, Long Lake thumbnail

· 17:46

How do you diffuse AI into the real world? — Varun Shenoy, Long Lake

The deployment of advanced AI agents into real-world service industries is not merely a technological challenge but an operational one. Drawing parallels to the adoption of electricity and Ford's assembly line, the speaker argues that technology diffusion takes generations. Long Lake addresses this by acquiring and operating services businesses (e.g., property management) rather than selling software. Their approach focuses on building AI agents that move beyond simple 'co-pilots' to become autonomous 'co-workers,' leveraging proprietary ground truth data collected from messy, real-world tasks—a process requiring deep, physical co-design with the industry.

Key takeaways

  1. AI Diffusion Takes Generations 1:30

    The adoption of general-purpose technologies (GPTs) is slow. Just as electricity took decades to fully integrate into industries like Ford's, AI requires massive operational shifts—ripping out old processes and retraining staff—to achieve full diffusion. [1:30]

  2. The Value of Owning the Outcome 2:36

    Long Lake does not sell AI software; they acquire and operate services businesses (e.g., HOA, architecture). By being the operator/owner, they bear the risk when the AI fails, ensuring deep integration and accountability that external vendors cannot match. [2:36]

  3. The Progression from Co-pilot to Co-worker 6:18

    AI agents must progress through stages of autonomy: Co-pilot (simple RAG chatbot) $ ightarrow$ Synchronous Agent (real-time, two-way interaction) $ ightarrow$ Asynchronous Agent (background work, external triggers) $ ightarrow$ Long-running Agent $ ightarrow$ AI Co-worker (proactive partner). Achieving the co-worker requires earning the right to do more through iterative field deployment. [6:02]

Watch on YouTube Full article

How AI Agents Let GTM Teams Scale — Justin Joyce, Cloudflare thumbnail

· 19:15

How AI Agents Let GTM Teams Scale — Justin Joyce, Cloudflare

Justin Joyce details how Cloudflare utilizes a three-pillar approach powered by AI agents to solve scaling issues in traditional Go-To-Market (GTM) operations. The core problem—that manual processes and knowledge gaps limit efficiency—is addressed by building self-service capabilities, automating data analysis, and proactively pushing insights to users. This framework allows non-technical staff to query complex data using 'skill files' and enables reps to access expert-level information via an internal agentic workspace (Cloudflare OS).

Key takeaways

  1. The GTM Scaling Problem 5:20

    Traditional GTM processes suffer from the 'context gap' (switching between different customer calls) and the 'expert gap' (inconsistent execution among reps), leading to significant inefficiency. Back-office teams often rely on manual work in Excel or limited dashboards, which fail to meet all operational needs.

  2. Pillar 1: Scaling Analysis via Skill Files 10:53

    To empower non-SQL users, the team built role-specific 'skill files.' These allow business users to ask questions of data directly and quickly, eliminating bottlenecks that previously required a dedicated data engineer. This capability is used both for back-office analysis and building GTM skills.

  3. Pillar 2: Scaling Insight (Pushing the Story) 15:00

    Instead of forcing users to search dashboards, insights are proactively delivered through synthetic weekly summaries. This is achieved by transforming data dimensions (time, manager, metric) and using a multi-agent workflow: a first draft agent, a second reviewer agent for veracity checks, and a third 'tone agent' that crafts the message, ensuring risks and opportunities are weighted equally.

  4. Pillar 3: Self-Service via Cloudflare OS 17:15

    The internal tool, Cloudflare OS, is an agentic workspace running on Workers and Durable Objects. It allows GTM teams to perform complex tasks—such as building QBR decks, account plans, or forecast briefs—by interacting with curated expert skills without needing manual intervention.

Watch on YouTube Full article

What Are Large Database Models? AI for SQL Data thumbnail

· 13:55

What Are Large Database Models? AI for SQL Data

Large Database Models (LDMs) represent a significant advancement in applying AI to enterprise data by bringing semantic capabilities directly into SQL and relational databases. Unlike Large Language Models (LLMs), which train on general text, LDMs are trained specifically on selected tables or views within a structured database. This allows organizations to unlock the estimated 99% of critical business data—often locked behind encryption and access controls—without needing to move it.

Key takeaways

  1. LDM Functionality vs. Traditional SQL 2:15

    Traditional methods require data scientists to manually write rigid SQL filters (e.g., `where age is between 20 and 40`) and move data to an analytics platform, which is slow and expensive. LDMs use vector representations learned from co-occurring values across columns to perform semantic queries, eliminating the need for manual field selection or guessing constraints.

  2. Core LDM Capabilities 3:30

    LDMs enable advanced querying capabilities such as finding customer similarity (finding customers 'most similar' to a given ID), identifying unusual transactions (fraud detection), and exploring product relationships, all executed via standard SQL against the database itself.

  3. Commercial Availability 9:00

    IBM launched the first LDM-based database product, 'SQL Data Insights,' which ships as part of DB2 for ZOS. A follow-up version, 'SQL Data Insights Pro,' extends this approach to unstructured text and adds incremental model refresh.

Watch on YouTube Full article

Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI thumbnail

· 19:05

Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI

In an era of increasing compute scarcity—evidenced by rising H100 prices and skyrocketing token usage—data quality has emerged as the critical 'compute multiplier' for model training. The presentation outlines a systematic approach to data enhancement through four stages: Clean, Curate, Create, and Compose. By maximizing the signal per token (marginal information gain), organizations can achieve performance levels comparable to models trained with vastly more compute budgets. Practical applications include improving Vision Language Models (VLMs) and enhancing multilingual capabilities using proprietary or public datasets.

Key takeaways

  1. Compute Scarcity Drives Data Focus

    The availability of compute is becoming increasingly constrained, leading to market actions like Google capping Meta's Gemini usage and OpenAI selling token futures. This necessitates a shift in focus from raw compute power to data quality.

  2. Data Quality as Compute Multiplier 3:39

    Improving data quality allows for dramatically better performance (blue curve) compared to training with the same limited compute budget (gray curve), effectively simulating much larger compute investments.

  3. The Four C's of Data Enhancement 5:48

    Data improvement is achieved through a pipeline: Clean (heuristic filters, decontamination), Curate (quality classifiers, redundancy reduction), Create (synthetic data generation/rephrasing), and Compose (sequencing across multiple training stages).

  4. Cross-Lingual Benefits from Curation 15:24

    Curating English data can positively benefit non-English performance, demonstrating cross-lingual transfer. Similarly, curating non-English data benefits English performance.

Watch on YouTube Full article

Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute thumbnail

· 18:20

Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

The presentation outlines a vision for advanced AI agents that can continuously learn and adapt in real-world enterprise environments post-deployment. The core methodology involves an orchestrated training loop: interactions are captured by an orchestrator, processed through inference engines, graded, and the resulting data (graded chats/traces) is fed into a training engine to compute weight updates. Key challenges addressed include environment fidelity, preventing reward hacking, and transitioning from controlled, replayable synthetic environments to uncontrolled, off-policy production data.

Key takeaways

  1. The Progression of Agent Training

    Agent training moves from simple single-turn Q&A tasks (controlled by a dedicated training stack) to complex, multi-turn, long-horizon tasks that require offloading environment state outside the training stack.

  2. The Core RL Training Loop

    Training relies on an orchestrator driving rollouts, which sends prompts to a model and then passes results to a grader. The resulting graded chats are used by a training engine to compute weight updates for the inference engines.

  3. Addressing Real-World Data Challenges 15:45

    Replicating production environments is difficult due to issues like non-replayability and off-policy data. The future requires methods like automated data pipelines and qualitative feedback ingestion to learn from real interactions.

  4. The Vision: Self-Improving Agents

    The ultimate goal is a single deployment model that can interact across many different settings, continuously self-evaluating and computing weight updates from every interaction it has.

Watch on YouTube Full article

The Base Model Is Dead — Varun Singh, Arcee AI thumbnail

· 17:45

The Base Model Is Dead — Varun Singh, Arcee AI

The traditional paradigm of viewing a base language model as merely a reflection of the entire human web is obsolete. As LLMs advance toward complex reasoning and agentic behavior (e.g., interacting with software environments), the training focus shifts from raw web text to incorporating instruction data, synthetic reasoning traces, and post-training techniques earlier into the pre-training phase. This requires careful management of data mixes, load balancing coefficients (especially in MOEs), and establishing stable representations early on.

Key takeaways

  1. The Decline of Raw Web Text 2:10

    Historically, models like GPT-3 relied heavily on raw web scrapes (e.g., Common Crawl/WebText-2), which constituted up to 85% of the training data. Modern recipes show a significant decrease in web text's proportion, indicating that its value is diminishing relative to code and structured reasoning abilities.

  2. Shift from Knowledge Prior to Capability Prior 8:00

    The base model's role is changing from accumulating general world knowledge (the 'prior') to carrying the necessary prior for complex Reinforcement Learning (RL) tasks. RL is no longer a mere 'cherry on top,' but a core component that requires the base model to be prepared for advanced composition and reasoning.

  3. Synthetic Data Integration 11:20

    A key trend involves pulling post-training data (like SFT/Q&A chat data) and large-scale synthetic data back into the pre-training phase. This allows models to learn task representations, conversation shapes, and atomic skills from the very beginning.

Watch on YouTube Full article