What My Motion tells me about Your Pose: A Self-Supervised Monocular 3D Vehicle Detector

TL;DR

This paper introduces a self-supervised approach for monocular 3D vehicle detection that leverages monocular visual odometry for orientation estimation, reducing reliance on labeled data and improving generalization across datasets.

Contribution

It presents a self-supervised fine-tuning method for vehicle orientation estimation and an optimization-based 3D bounding box detector that does not require expensive labeled data.

Findings

Recovered up to 70% of supervised performance when transitioning datasets

Enabled self-training of 3D vehicle detection from large monocular datasets

Demonstrated effective self-supervised fine-tuning across different domains

Abstract

The estimation of the orientation of an observed vehicle relative to an Autonomous Vehicle (AV) from monocular camera data is an important building block in estimating its 6 DoF pose. Current Deep Learning based solutions for placing a 3D bounding box around this observed vehicle are data hungry and do not generalize well. In this paper, we demonstrate the use of monocular visual odometry for the self-supervised fine-tuning of a model for orientation estimation pre-trained on a reference domain. Specifically, while transitioning from a virtual dataset (vKITTI) to nuScenes, we recover up to 70% of the performance of a fully supervised method. We subsequently demonstrate an optimization-based monocular 3D bounding box detector built on top of the self-supervised vehicle orientation estimator without the requirement of expensive labeled data. This allows 3D vehicle detection algorithms to be self-trained from large amounts of monocular camera data from existing commercial vehicle fleets.