Synthetic Data Generation — The Video Signal

Sep 3, 2026 · 50:35

How To Build And Evaluate Search Agents

This talk details three critical areas for building and evaluating advanced search agents: evaluation benchmarks, synthetic data generation, and trajectory analysis. The speaker introduces ORBIT, a novel pipeline for creating complex, multi-hop questions without paid APIs or labeled data. For evaluation, the presentation highlights BrowseComp-Plus as a reproducible benchmark that converts QA tasks into open retrieval problems. Key findings emphasize that both the retriever model and the LLM contribute significantly to accuracy, but efficiency (fewer search calls/turns) is crucial for practical deployment. Finally, trajectory analysis tools like Hawkeye are presented as essential for diagnosing agent failures by visualizing query provenance, topic shifts, and repetition.

Key takeaways

Search Agents vs. RAG Architecture 5:45
Agentic search differs from naive Retrieval-Augmented Generation (RAG) because the LLM acts as the main driver, iteratively interacting with a retrieval tool to refine queries before generating a final answer. This iterative process is key.
ORBIT for Synthetic Data 17:25
The ORBIT pipeline generates complex, multi-hop questions by describing an entity's properties without naming it (inverting the question). It verifies every question by requiring the agent to confirm claims against source documents.
BrowseComp-Plus Reproducibility 22:20
To improve reproducibility, BrowseComp-Plus converts QA tasks into open retrieval problems. This involves providing a web corpus (including hard negatives) and human relevance judgments for each query/answer pair.
Efficiency vs. Accuracy in Search Agents 29:20
While high Exact Match (EM) accuracy is good, efficiency—measured by the number of search calls or turns—is equally important. A model with fewer search calls but comparable accuracy is often preferred.
Trajectory Analysis Importance 41:00
Relying solely on EM accuracy is insufficient. Analyzing agent trajectories helps identify issues like query repetition, topic shifts, and inefficient looping (e.g., a model taking many turns without reaching the answer).

Watch on YouTube Full article

Aug 19, 2026 · 16:46

Don’t be data poor — Anuj Iravane, Anterior

The talk addresses the critical problem of 'data poverty' in highly regulated domains like healthcare, where the most valuable data (Patient Health Information or PHI) is ephemeral and legally prohibited from being retained, anonymized, or derived for dataset creation. The core solution presented is synthetic data generation. This process involves reversing the standard inference workflow—starting by sampling a desired label and reasoning trace, and then generating the necessary unstructured medical record that would have produced it. The resulting pipeline uses an LLM-based, coarse-to-fine approach, ensuring high fidelity while maintaining domain expert control.

Key takeaways

Reverse Inference for Data Generation 5:20
Instead of running the forward task (Unstructured Data + Policy $\rightarrow$ Label), the method reverses this by sampling a label and a reasoning trace first, then generating the input data that supports it. This circumvents the diversity problem inherent in standard LLM generation.
Domain Expert Ownership (Human-in-the-Loop) 11:30
To ensure generated data is useful, domain experts (clinicians) must own the pipeline. This is achieved by enabling them to interject at any point in generation and modeling the entire workflow as a skills-based system running on an agent harness.
Synthetic Data Fidelity 14:35
The generated data can be highly accurate, with early results showing that in a blind review, clinicians were only able to distinguish synthetic from real records about 60% of the time.

Watch on YouTube Full article

Jul 29, 2026 · 21:09

Persona Engineering: A Field Guide to AI Synthetic Personas — Ishan Anand, InsightSciences.ai

Synthetic personas, powered by LLMs, offer a powerful method for simulating human behavior in market research. However, they are not ground truth and must be treated like weather forecasts—predictive models operating within defined regimes. The talk emphasizes that accuracy is often misleading; failure modes include the model inventing latent confounders (e.g., using price as a proxy for product quality) and extreme prompt sensitivity to variable ordering. Robust validation requires measuring the full distribution shape, not just average agreement.

Key takeaways

Treat Synthetic Personas as Forecasts, Not Facts
Synthetic personas are bounded systems; they predict potential outcomes but cannot guarantee absolute truth. Validation must involve comparing distributions against real-world data (the 'noise floor') rather than aiming for a perfect match.
Beware of Latent Confounders 9:52
LLMs can invent or infer confounders when context is missing. Poorly grounded prompts allow the model to 'play improv,' leading to skewed results (e.g., an inverted U-shaped purchase probability curve where price increases lead to increased purchase likelihood). Rich prompting must specify personality, context, and study construction.
Focus on Distribution Shape, Not Just Average Accuracy 20:49
When evaluating performance, focus on metrics that capture the entire shape similarity of the distribution (e.g., using correlation and specific shape metrics). LLMs often lose variation details when averaging results, which is a critical failure mode.
Behavior vs. Stated Attitude 15:07
LLMs are trained on what people *say* (text/surveys) and perform better predicting stated attitudes than actual behaviors or actions, which require more complex transcription into text.

Watch on YouTube Full article

Jul 26, 2026 · 17:31

The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

The presentation details how poolside scaled their LLM training from smaller models (Laguna M/XS) to a massive 118 billion parameter model for agentic coding. The core strategy involves creating sophisticated synthetic data pipelines—which complement organic data by exposing implicit rationale and structure—and implementing extreme rigor in the pre-training process. Key technical challenges addressed include catching silent failures like broken GPUs, numerical precision loss during tensor parallel accumulation (BF16 to FP32 fixes), and race conditions via weight hashing checks. The resulting model, Laguna S, demonstrates competitive performance against industry leaders on coding benchmarks.

Key takeaways

Synthetic Data for Feature Extraction 10:37
Synthetic data is used not to replace organic data but to complement it by extracting and projecting implicitly hidden features (e.g., implicit rationale, planning, structure). Pipelines are modular, allowing complex tasks to be broken down into stages like multi-stage workflows, cross-domain porting (e.g., math problems to code), or multi-turn chats.
Rigorous Training Validation: Trust Nothing 15:03
To ensure correctness at scale, the team implemented model replica hashing checks; if hashes over weights do not match across distributed replicas, training is immediately halted. Failures observed include silent data corruption from broken GPUs and gradient corruption due to race conditions.
Scaling Success: The 118B Parameter Model
The final model, Laguna S (118B parameters), was trained on 30 trillion tokens across 4,000 GPUs. Early results show strong performance in agentic coding benchmarks (e.g., SWEETbench), outperforming models like GLM 4.5 Air.

Watch on YouTube Full article