Black-Box Test-Time Shape REFINEment for Single View 3D Reconstruction

TL;DR

This paper introduces REFINE, a postprocessing mesh refinement method that improves single view 3D reconstruction accuracy and consistency by optimizing meshes at test time, applicable to any black-box model.

Contribution

The paper presents a universal postprocessing refinement step that enhances existing 3D reconstruction methods by optimizing meshes for view consistency and domain generalization.

Findings

Achieves state-of-the-art performance on benchmark datasets.

Reduces domain gap and improves mesh-view consistency.

Applicable to any black-box 3D reconstruction method.

Abstract

Much recent progress has been made in reconstructing the 3D shape of an object from an image of it, i.e. single view 3D reconstruction. However, it has been suggested that current methods simply adopt a "nearest-neighbor" strategy, instead of genuinely understanding the shape behind the input image. In this paper, we rigorously show that for many state of the art methods, this issue manifests as (1) inconsistencies between coarse reconstructions and input images, and (2) inability to generalize across domains. We thus propose REFINE, a postprocessing mesh refinement step that can be easily integrated into the pipeline of any black-box method in the literature. At test time, REFINE optimizes a network per mesh instance, to encourage consistency between the mesh and the given object view. This, along with a novel combination of regularizing losses, reduces the domain gap and achieves state of the art performance. We believe that this novel paradigm is an important step towards robust, accurate reconstructions, remaining relevant as new reconstruction networks are introduced.