Robust Object Detection with Pseudo Labels from VLMs using Per-Object Co-teaching

TL;DR

This paper presents a novel per-object co-teaching pipeline that leverages vision-language models to generate pseudo-labels for training real-time, high-performance object detectors in autonomous driving, reducing reliance on manual annotations.

Contribution

The work introduces a per-object co-teaching strategy that filters noisy VLM-generated labels at the object level, improving detection accuracy and robustness.

Findings

Outperforms baseline YOLOv5m with 46.61% mAP@0.5 on KITTI

Adding 10% ground truth labels boosts mAP@0.5 to 57.97%

Achieves real-time detection suitable for autonomous driving

Abstract

Foundation models, especially vision-language models (VLMs), offer compelling zero-shot object detection for applications like autonomous driving, a domain where manual labelling is prohibitively expensive. However, their detection latency and tendency to hallucinate predictions render them unsuitable for direct deployment. This work introduces a novel pipeline that addresses this challenge by leveraging VLMs to automatically generate pseudo-labels for training efficient, real-time object detectors. Our key innovation is a per-object co-teaching-based training strategy that mitigates the inherent noise in VLM-generated labels. The proposed per-object coteaching approach filters noisy bounding boxes from training instead of filtering the entire image. Specifically, two YOLO models learn collaboratively, filtering out unreliable boxes from each mini-batch based on their peers' per-object loss values. Overall, our pipeline provides an efficient, robust, and scalable approach to train high-performance object detectors for autonomous driving, significantly reducing reliance on costly human annotation. Experimental results on the KITTI dataset demonstrate that our method outperforms a baseline YOLOv5m model, achieving a significant mAP@0.5 boost ($31.12\%$ to $46.61\%$) while maintaining real-time detection latency. Furthermore, we show that supplementing our pseudo-labelled data with a small fraction of ground truth labels ($10\%$) leads to further performance gains, reaching $57.97\%$ mAP@0.5 on the KITTI dataset. We observe similar performance improvements for the ACDC and BDD100k datasets.