Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face

Summary

The talk argues that developers should stop using Vision Language Models (VLMs) directly for real-time computer vision tasks, as they are often insufficient for high-performance, low-latency applications. Instead, the speaker introduces a comprehensive toolkit and methodology for building robust, task-specific detectors (like RFDTR) using Apache 2.0 licensed models. A key innovation is the 'vibe training' pipeline, which leverages VLMs as labelers and judges to efficiently train specialized models when only unlabeled images are available. The entire process is designed to be scalable and cost-effective, running on cloud infrastructure.

Technical details

• RFDTR Detector Training 341s

  The pipeline involves: 1) Labeling data with a VLM (e.g., Q1 3.5B). 2) Passing labeled data to two smaller VLM judges (e.g., Gemma 4 E4B and LFM 2.5VL). 3) Merging judgments based on minimum agreement (not consensus). 4) Training the specialized detector (RFDTR medium or large).

• Infrastructure and Cost 341s

  The toolkit supports long-horizon tasks using cloud infrastructure (Hugging Face jobs, serverless routing/inference providers) and costs approximately $3-$4 to run the entire training pipeline.

• Zero-Shot Segmentation 560s

  The speaker highlights models like Falcon Perception (600M parameters, Apache 2.0) which can perform open-ended reference-based segmentation (e.g., 'the red car next to the orange car').

• Agent Behavior Flaws 460s

  Coding agents, even advanced ones (e.g., OPUS 4.8), can exhibit common-sense failures, such as horizontally flipping traffic signs or jittering traffic light colors, requiring manual patching.

Mentioned resources

• Web Vision Toolkit (Toolkit)
• RFDTR (Model/Detector)
• Falcon Perception (Model)
• Hugging Face Skills (Platform Feature)

Channel & topics

• AI Engineer
• Computer Vision
• Vision Language Models (VLM)
• Object Detection
• Segmentation
• Machine Learning Pipelines
• Model Licensing
• Web Vision Toolkit
• RFDTR
• Falcon Perception
• Hugging Face Skills