Mirror3D: Depth Refinement for Mirror Surfaces

TL;DR

Mirror3D introduces a new dataset and a depth refinement method specifically targeting errors caused by mirror surfaces in 3D reconstruction, improving accuracy across various depth inputs.

Contribution

The paper presents Mirror3DNet, a novel module that estimates mirror planes from RGB and refines depth data, addressing a key challenge in 3D reconstruction involving mirrors.

Findings

Mirror3DNet significantly reduces depth errors on mirror surfaces.

The Mirror3D dataset contains over 7,000 mirror instances across three datasets.

The method improves depth accuracy for raw sensor and estimated depths.

Abstract

Despite recent progress in depth sensing and 3D reconstruction, mirror surfaces are a significant source of errors. To address this problem, we create the Mirror3D dataset: a 3D mirror plane dataset based on three RGBD datasets (Matterport3D, NYUv2 and ScanNet) containing 7,011 mirror instance masks and 3D planes. We then develop Mirror3DNet: a module that refines raw sensor depth or estimated depth to correct errors on mirror surfaces. Our key idea is to estimate the 3D mirror plane based on RGB input and surrounding depth context, and use this estimate to directly regress mirror surface depth. Our experiments show that Mirror3DNet significantly mitigates errors from a variety of input depth data, including raw sensor depth and depth estimation or completion methods.