On the generalization of learning-based 3D reconstruction

TL;DR

This paper investigates how model architecture biases affect the ability of learning-based monocular 3D reconstruction methods to generalize to unseen object categories, proposing mechanisms to improve this generalization.

Contribution

The paper identifies key inductive biases influencing generalization and introduces methods to enforce these biases, leading to improved performance on standard benchmarks.

Findings

Achieves state-of-the-art results on ShapeNet benchmark

Identifies three key inductive biases affecting generalization

Proposes mechanisms to enforce biases and improve unseen category reconstruction

Abstract

State-of-the-art learning-based monocular 3D reconstruction methods learn priors over object categories on the training set, and as a result struggle to achieve reasonable generalization to object categories unseen during training. In this paper we study the inductive biases encoded in the model architecture that impact the generalization of learning-based 3D reconstruction methods. We find that 3 inductive biases impact performance: the spatial extent of the encoder, the use of the underlying geometry of the scene to describe point features, and the mechanism to aggregate information from multiple views. Additionally, we propose mechanisms to enforce those inductive biases: a point representation that is aware of camera position, and a variance cost to aggregate information across views. Our model achieves state-of-the-art results on the standard ShapeNet 3D reconstruction benchmark in various settings.