Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint

TL;DR

This paper introduces a novel convex relaxation approach for dense semantic 3D reconstruction that incorporates a visibility consistency constraint, improving accuracy and enabling the reconstruction of thin objects.

Contribution

It presents a new convex formulation with a non-convex visibility constraint and a majorize-minimize optimization strategy for semantic 3D reconstruction.

Findings

Achieved state-of-the-art results on Middlebury multi-view stereo benchmark.

Successfully reconstructed thin objects without specialized algorithms.

Significantly improved semantic 3D reconstruction quality.

Abstract

We propose an approach for dense semantic 3D reconstruction which uses a data term that is defined as potentials over viewing rays, combined with continuous surface area penalization. Our formulation is a convex relaxation which we augment with a crucial non-convex constraint that ensures exact handling of visibility. To tackle the non-convex minimization problem, we propose a majorize-minimize type strategy which converges to a critical point. We demonstrate the benefits of using the non-convex constraint experimentally. For the geometry-only case, we set a new state of the art on two datasets of the commonly used Middlebury multi-view stereo benchmark. Moreover, our general-purpose formulation directly reconstructs thin objects, which are usually treated with specialized algorithms. A qualitative evaluation on the dense semantic 3D reconstruction task shows that we improve significantly over previous methods.