Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face
The talk argues that developers should stop using Vision Language Models (VLMs) directly for real-time computer vision tasks, as they are often insufficient for high-performance, low-latency applications. Instead, the speaker introduces a comprehensive toolkit and methodology for building robust, task-specific detectors (like RFDTR) using Apache 2.0 licensed models. A key innovation is the 'vibe training' pipeline, which leverages VLMs as labelers and judges to efficiently train specialized models when only unlabeled images are available. The entire process is designed to be scalable and cost-effective, running on cloud infrastructure.
Key takeaways
-
VLM Limitations for Real-Time CV
VLMs are unsuitable for real-time applications; specialized detectors (e.g., RFDTR) running on hardware like a toaster can achieve 30-40 FPS, outperforming VLMs.
-
Licensing Compliance
2:31
Developers must prioritize using Apache 2.0 licensed models and avoid deploying popular detectors (like YOLO) without checking their copyleft licenses (e.g., AGPL 3.0).
-
Vibe Training Pipeline
5:41
A novel pipeline uses a VLM as a labeler and a second VLM as a judge to annotate and evaluate data, enabling the training of specialized detectors (RFDTR) from unlabeled datasets.