# Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face

## Executive summary

The talk argues that developers should stop using Vision Language Models (VLMs) directly for real-time computer vision tasks, as they are often insufficient for high-performance, low-latency applications. Instead, the speaker introduces a comprehensive toolkit and methodology for building robust, task-specific detectors (like RFDTR) using Apache 2.0 licensed models. A key innovation is the 'vibe training' pipeline, which leverages VLMs as labelers and judges to efficiently train specialized models when only unlabeled images are available. The entire process is designed to be scalable and cost-effective, running on cloud infrastructure.

## Key takeaways

- VLM Limitations for Real-Time CV: VLMs are unsuitable for real-time applications; specialized detectors (e.g., RFDTR) running on hardware like a toaster can achieve 30-40 FPS, outperforming VLMs.
- Licensing Compliance: Developers must prioritize using Apache 2.0 licensed models and avoid deploying popular detectors (like YOLO) without checking their copyleft licenses (e.g., AGPL 3.0).
- Vibe Training Pipeline: A novel pipeline uses a VLM as a labeler and a second VLM as a judge to annotate and evaluate data, enabling the training of specialized detectors (RFDTR) from unlabeled datasets.

## Technical details

- RFDTR Detector Training: The pipeline involves: 1) Labeling data with a VLM (e.g., Q1 3.5B). 2) Passing labeled data to two smaller VLM judges (e.g., Gemma 4 E4B and LFM 2.5VL). 3) Merging judgments based on minimum agreement (not consensus). 4) Training the specialized detector (RFDTR medium or large).
- Infrastructure and Cost: The toolkit supports long-horizon tasks using cloud infrastructure (Hugging Face jobs, serverless routing/inference providers) and costs approximately $3-$4 to run the entire training pipeline.
- Zero-Shot Segmentation: The speaker highlights models like Falcon Perception (600M parameters, Apache 2.0) which can perform open-ended reference-based segmentation (e.g., 'the red car next to the orange car').
- Agent Behavior Flaws: Coding agents, even advanced ones (e.g., OPUS 4.8), can exhibit common-sense failures, such as horizontally flipping traffic signs or jittering traffic light colors, requiring manual patching.

## Practical implications

- For building CV applications, focus on specialized, licensed models (Apache 2.0) rather than general VLMs for real-time performance.
- Implement a structured training pipeline (VLM labeler $\rightarrow$ VLM judge $\rightarrow$ Detector training) to handle unlabeled data at scale.
- Use cloud infrastructure (Hugging Face jobs) for scalable, cost-effective execution of complex, multi-stage CV pipelines.
- When developing with agents, anticipate and patch common-sense failures (e.g., geometric transformations) in the agent's output.

## Topics

Computer Vision, Vision Language Models (VLM), Object Detection, Segmentation, Machine Learning Pipelines, Model Licensing, Web Vision Toolkit, RFDTR, Falcon Perception, Hugging Face Skills

Source: https://www.youtube.com/watch?v=dKcTBQzR7jI