MAtCha Gaussians: Atlas of Charts for High-Quality Geometry and Photorealism From Sparse Views

TL;DR

MAtCha Gaussians introduces a novel scene modeling approach that combines high-quality 3D surface mesh recovery with photorealistic view synthesis from sparse views, using Gaussian surfels on an Atlas of Charts.

Contribution

The paper proposes a new appearance model that integrates mesh and neural rendering, refining monocular depth estimates with Gaussian surfels for improved geometry and photorealism.

Findings

State-of-the-art surface reconstruction quality

High photorealism with fewer input views

Reduced computational time compared to existing methods

Abstract

We present a novel appearance model that simultaneously realizes explicit high-quality 3D surface mesh recovery and photorealistic novel view synthesis from sparse view samples. Our key idea is to model the underlying scene geometry Mesh as an Atlas of Charts which we render with 2D Gaussian surfels (MAtCha Gaussians). MAtCha distills high-frequency scene surface details from an off-the-shelf monocular depth estimator and refines it through Gaussian surfel rendering. The Gaussian surfels are attached to the charts on the fly, satisfying photorealism of neural volumetric rendering and crisp geometry of a mesh model, i.e., two seemingly contradicting goals in a single model. At the core of MAtCha lies a novel neural deformation model and a structure loss that preserve the fine surface details distilled from learned monocular depths while addressing their fundamental scale ambiguities. Results of extensive experimental validation demonstrate MAtCha's state-of-the-art quality of surface reconstruction and photorealism on-par with top contenders but with dramatic reduction in the number of input views and computational time. We believe MAtCha will serve as a foundational tool for any visual application in vision, graphics, and robotics that require explicit geometry in addition to photorealism. Our project page is the following: https://anttwo.github.io/matcha/