Meshed Up: Learnt Error Correction in 3D Reconstructions

TL;DR

This paper introduces a machine learning method to detect and quantify errors in 3D mesh reconstructions, leading to improved depth accuracy by correcting errors in lower-quality reconstructions.

Contribution

We develop a deep learning approach that predicts error magnitude and direction in 3D meshes, enabling enhanced correction of reconstruction inaccuracies.

Findings

Improves depth reconstruction RMSE by up to 10%

Effectively identifies and quantifies errors in 3D meshes

Enhances 2D inverse-depth image quality

Abstract

Dense reconstructions often contain errors that prior work has so far minimised using high quality sensors and regularising the output. Nevertheless, errors still persist. This paper proposes a machine learning technique to identify errors in three dimensional (3D) meshes. Beyond simply identifying errors, our method quantifies both the magnitude and the direction of depth estimate errors when viewing the scene. This enables us to improve the reconstruction accuracy. We train a suitably deep network architecture with two 3D meshes: a high-quality laser reconstruction, and a lower quality stereo image reconstruction. The network predicts the amount of error in the lower quality reconstruction with respect to the high-quality one, having only view the former through its input. We evaluate our approach by correcting two-dimensional (2D) inverse-depth images extracted from the 3D model, and show that our method improves the quality of these depth reconstructions by up to a relative 10% RMSE.