Co-training for Deep Object Detection: Comparing Single-modal and Multi-modal Approaches

TL;DR

This paper explores co-training methods for deep object detection, comparing single-modal (appearance) and multi-modal (appearance and depth) approaches, demonstrating multi-modal's advantages especially under domain shift conditions.

Contribution

It introduces a multi-modal co-training framework for self-labeling object bounding boxes and compares its effectiveness against single-modal methods in various domain shift scenarios.

Findings

Multi-modal co-training outperforms single-modal in standard SSL settings.

Multi-modal approach is more robust under domain shift, especially with GAN-based translation.

Using off-the-shelf depth estimation models suffices without retraining on translated images.

Abstract

Top-performing computer vision models are powered by convolutional neural networks (CNNs). Training an accurate CNN highly depends on both the raw sensor data and their associated ground truth (GT). Collecting such GT is usually done through human labeling, which is time-consuming and does not scale as we wish. This data labeling bottleneck may be intensified due to domain shifts among image sensors, which could force per-sensor data labeling. In this paper, we focus on the use of co-training, a semi-supervised learning (SSL) method, for obtaining self-labeled object bounding boxes (BBs), i.e., the GT to train deep object detectors. In particular, we assess the goodness of multi-modal co-training by relying on two different views of an image, namely, appearance (RGB) and estimated depth (D). Moreover, we compare appearance-based single-modal co-training with multi-modal. Our results suggest that in a standard SSL setting (no domain shift, a few human-labeled data) and under virtual-to-real domain shift (many virtual-world labeled data, no human-labeled data) multi-modal co-training outperforms single-modal. In the latter case, by performing GAN-based domain translation both co-training modalities are on pair; at least, when using an off-the-shelf depth estimation model not specifically trained on the translated images.