Inside Factory Playback: How Tulip Engineered a Replayable Factory powered by NVIDIA
Summary
Factory Playback is an operational platform designed to create a unified, replayable record of manufacturing processes by integrating disparate data sources. It leverages advanced AI, specifically NVIDIA Cosmos and NVIDIA Metropolis VSS, to process raw video footage, machine telemetry, and operator workflows. This capability moves manufacturers beyond merely reconstructing events from isolated dashboards, allowing them to perform deep root cause analysis and identify process inefficiencies by treating the factory floor as a searchable, queryable data asset.
Key takeaways
-
The Problem of Siloed Data
1:45
Traditional manufacturing software systems are highly isolated, making it difficult and expensive to integrate different systems (e.g., ERPs, PLMs, sensors) to gain a unified view of the shop floor.
-
Factory Playback's Core Function
2:40
The platform enriches digital touch points by giving 'eyes' to manufacturing systems, connecting cameras and using Vision Language Models (VLMs) to process video continuously for real-time alerts and retrospective analysis.
-
AI for Searchable Video (VSS)
3:20
NVIDIA Metropolis VSS turns raw video footage into an agent that can be queried, allowing users to ask questions about the footage (e.g., 'Show me every time an operator reached into a fixture while the machine was cycling') and receive summarized, searchable answers.
-
Hybrid, On-Premise Architecture
4:40
Factory Playback operates on a hybrid cloud model. Video processing and storage occur entirely on-premises (using an on-board server with NVIDIA GPUs) to maintain data privacy and minimize bandwidth, while high-level events are pushed to the Tulip cloud for consumption by automations.
Technical details
-
Tulip Operational Platform (U)
120s
U is a cloud-based platform designed to allow manufacturing users to build bespoke solutions by connecting various systems (sensors, machines, ERPs, PLMs) that are often siloed in traditional factory settings.
-
NVIDIA Metropolis VSS Blueprint
200s
This reference architecture enables significant Vision AI agents by combining multiple models: a Vision Language Model (VLM), an NLM (for reasoning and searchability), and Computer Vision (for fast detection/tracking). It supports deep search, verified alerts, and event summarization.
-
NVIDIA Cosmos Model
220s
Cosmos is an open frontier omni model for physical AI. It can reason about video, simulate worlds, generate synthetic data (useful for rare failure detection), and drive spatial policies for robotics. It is fully multimodal and available in multiple sizes (4B to 64B parameters).
-
System Architecture and Connectivity
280s
The playback system runs on an on-premises K3S stack, connected to local camera streams and MQTT brokers. This allows for the ingestion of time series data (telemetry) alongside video data, enabling correlation between machine alerts and physical events.
Mentioned resources
- Tulip technical blog
- NVIDIA Cosmos platform
- NVIDIA Metropolis VSS blueprint
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.