NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localization

TL;DR

NeurOCS introduces a novel framework that leverages differentiable rendering and shape priors to improve monocular 3D object localization in driving scenes, achieving state-of-the-art results on KITTI.

Contribution

It proposes a new method for learning dense 3D object coordinates using only masks and 3D boxes, without requiring high-quality supervision or CAD models.

Findings

Achieves 1st place on KITTI-Object benchmark among monocular methods.

Effectively learns category-level shape priors from real scenes.

Improves dense 3D localization accuracy in driving environments.

Abstract

Monocular 3D object localization in driving scenes is a crucial task, but challenging due to its ill-posed nature. Estimating 3D coordinates for each pixel on the object surface holds great potential as it provides dense 2D-3D geometric constraints for the underlying PnP problem. However, high-quality ground truth supervision is not available in driving scenes due to sparsity and various artifacts of Lidar data, as well as the practical infeasibility of collecting per-instance CAD models. In this work, we present NeurOCS, a framework that uses instance masks and 3D boxes as input to learn 3D object shapes by means of differentiable rendering, which further serves as supervision for learning dense object coordinates. Our approach rests on insights in learning a category-level shape prior directly from real driving scenes, while properly handling single-view ambiguities. Furthermore, we study and make critical design choices to learn object coordinates more effectively from an object-centric view. Altogether, our framework leads to new state-of-the-art in monocular 3D localization that ranks 1st on the KITTI-Object benchmark among published monocular methods.