HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction

TL;DR

HiSplat introduces a hierarchical 3D Gaussian Splatting framework that improves 3D scene reconstruction and novel view synthesis from sparse views by capturing both large-scale structures and fine details through a coarse-to-fine approach.

Contribution

The paper proposes a hierarchical Gaussian Splatting method with inter-scale interaction modules, enhancing reconstruction quality and generalization over prior single-scale approaches.

Findings

Significantly improves reconstruction quality over prior methods.

Enhances cross-dataset generalization in 3D scene reconstruction.

Effective in synthesizing novel views from only two input images.

Abstract

Reconstructing 3D scenes from multiple viewpoints is a fundamental task in stereo vision. Recently, advances in generalizable 3D Gaussian Splatting have enabled high-quality novel view synthesis for unseen scenes from sparse input views by feed-forward predicting per-pixel Gaussian parameters without extra optimization. However, existing methods typically generate single-scale 3D Gaussians, which lack representation of both large-scale structure and texture details, resulting in mislocation and artefacts. In this paper, we propose a novel framework, HiSplat, which introduces a hierarchical manner in generalizable 3D Gaussian Splatting to construct hierarchical 3D Gaussians via a coarse-to-fine strategy. Specifically, HiSplat generates large coarse-grained Gaussians to capture large-scale structures, followed by fine-grained Gaussians to enhance delicate texture details. To promote inter-scale interactions, we propose an Error Aware Module for Gaussian compensation and a Modulating Fusion Module for Gaussian repair. Our method achieves joint optimization of hierarchical representations, allowing for novel view synthesis using only two-view reference images. Comprehensive experiments on various datasets demonstrate that HiSplat significantly enhances reconstruction quality and cross-dataset generalization compared to prior single-scale methods. The corresponding ablation study and analysis of different-scale 3D Gaussians reveal the mechanism behind the effectiveness. Project website: https://open3dvlab.github.io/HiSplat/