# Inside Factory Playback: How Tulip Engineered a Replayable Factory powered by NVIDIA

## Executive summary

Factory Playback is an operational platform designed to create a unified, replayable record of manufacturing processes by integrating disparate data sources. It leverages advanced AI, specifically NVIDIA Cosmos and NVIDIA Metropolis VSS, to process raw video footage, machine telemetry, and operator workflows. This capability moves manufacturers beyond merely reconstructing events from isolated dashboards, allowing them to perform deep root cause analysis and identify process inefficiencies by treating the factory floor as a searchable, queryable data asset.

## Key takeaways

- The Problem of Siloed Data: Traditional manufacturing software systems are highly isolated, making it difficult and expensive to integrate different systems (e.g., ERPs, PLMs, sensors) to gain a unified view of the shop floor.
- Factory Playback's Core Function: The platform enriches digital touch points by giving 'eyes' to manufacturing systems, connecting cameras and using Vision Language Models (VLMs) to process video continuously for real-time alerts and retrospective analysis.
- AI for Searchable Video (VSS): NVIDIA Metropolis VSS turns raw video footage into an agent that can be queried, allowing users to ask questions about the footage (e.g., 'Show me every time an operator reached into a fixture while the machine was cycling') and receive summarized, searchable answers.
- Hybrid, On-Premise Architecture: Factory Playback operates on a hybrid cloud model. Video processing and storage occur entirely on-premises (using an on-board server with NVIDIA GPUs) to maintain data privacy and minimize bandwidth, while high-level events are pushed to the Tulip cloud for consumption by automations.

## Technical details

- Tulip Operational Platform (U): U is a cloud-based platform designed to allow manufacturing users to build bespoke solutions by connecting various systems (sensors, machines, ERPs, PLMs) that are often siloed in traditional factory settings.
- NVIDIA Metropolis VSS Blueprint: This reference architecture enables significant Vision AI agents by combining multiple models: a Vision Language Model (VLM), an NLM (for reasoning and searchability), and Computer Vision (for fast detection/tracking). It supports deep search, verified alerts, and event summarization.
- NVIDIA Cosmos Model: Cosmos is an open frontier omni model for physical AI. It can reason about video, simulate worlds, generate synthetic data (useful for rare failure detection), and drive spatial policies for robotics. It is fully multimodal and available in multiple sizes (4B to 64B parameters).
- System Architecture and Connectivity: The playback system runs on an on-premises K3S stack, connected to local camera streams and MQTT brokers. This allows for the ingestion of time series data (telemetry) alongside video data, enabling correlation between machine alerts and physical events.

## Practical implications

- Enables root cause analysis by correlating historical video data with machine alerts and operational logs, answering 'why' a failure occurred, not just 'what' happened.
- Improves safety and quality control by building vision pipelines that detect deviations (e.g., unauthorized access, improper PPE, quality issues) and trigger automated alerts.
- Reduces reliance on manual data scrubbing by turning raw video into a structured, searchable database of events, significantly accelerating investigation time.

## Topics

Industrial AI, Computer Vision, Edge Computing, Manufacturing Automation, Data Integration, Time Series Analysis, Tulip technical blog, NVIDIA Cosmos platform, NVIDIA Metropolis VSS blueprint

Source: https://www.youtube.com/watch?v=AzMzbF8lJ-8