Bayes3D: fast learning and inference in structured generative models of 3D objects and scenes

TL;DR

Bayes3D is a perception system that rapidly learns and infers 3D object shapes and scene composition with high accuracy and efficiency, even in cluttered environments, using a hierarchical Bayesian model and GPU acceleration.

Contribution

It introduces a novel hierarchical Bayesian model and GPU-accelerated inference algorithm for fast, uncertainty-aware 3D scene understanding and object recognition.

Findings

Learns 3D object models from few views

Recognizes objects more robustly than neural baselines

Tracks 3D objects faster than real time on GPU

Abstract

Robots cannot yet match humans' ability to rapidly learn the shapes of novel 3D objects and recognize them robustly despite clutter and occlusion. We present Bayes3D, an uncertainty-aware perception system for structured 3D scenes, that reports accurate posterior uncertainty over 3D object shape, pose, and scene composition in the presence of clutter and occlusion. Bayes3D delivers these capabilities via a novel hierarchical Bayesian model for 3D scenes and a GPU-accelerated coarse-to-fine sequential Monte Carlo algorithm. Quantitative experiments show that Bayes3D can learn 3D models of novel objects from just a handful of views, recognizing them more robustly and with orders of magnitude less training data than neural baselines, and tracking 3D objects faster than real time on a single GPU. We also demonstrate that Bayes3D learns complex 3D object models and accurately infers 3D scene composition when used on a Panda robot in a tabletop scenario.