Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction

TL;DR

This paper introduces Neural Parametric Gaussians (NPGs), a two-stage method for high-quality, consistent non-rigid object reconstruction from monocular videos, even with limited multi-view cues.

Contribution

The paper proposes Neural Parametric Gaussians, combining a neural deformation model with local 3D Gaussian representations, to improve non-rigid reconstruction quality and view consistency.

Findings

NPGs outperform previous methods in challenging scenarios.

Achieves high-quality, photo-realistic reconstructions.

Provides consistent novel views from limited multi-view data.

Abstract

Reconstructing dynamic objects from monocular videos is a severely underconstrained and challenging problem, and recent work has approached it in various directions. However, owing to the ill-posed nature of this problem, there has been no solution that can provide consistent, high-quality novel views from camera positions that are significantly different from the training views. In this work, we introduce Neural Parametric Gaussians (NPGs) to take on this challenge by imposing a two-stage approach: first, we fit a low-rank neural deformation model, which then is used as regularization for non-rigid reconstruction in the second stage. The first stage learns the object's deformations such that it preserves consistency in novel views. The second stage obtains high reconstruction quality by optimizing 3D Gaussians that are driven by the coarse model. To this end, we introduce a local 3D Gaussian representation, where temporally shared Gaussians are anchored in and deformed by local oriented volumes. The resulting combined model can be rendered as radiance fields, resulting in high-quality photo-realistic reconstructions of the non-rigidly deforming objects. We demonstrate that NPGs achieve superior results compared to previous works, especially in challenging scenarios with few multi-view cues.