Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting

TL;DR

This paper introduces MM-GS, a hierarchical 3D Gaussian Splatting framework for reconstructing dynamic multi-human and multi-object scenes from sparse views, addressing occlusion and interaction modeling challenges.

Contribution

The paper presents a novel hierarchical approach combining multi-view fusion and scene graph reasoning to improve multi-instance 3D scene reconstruction.

Findings

Outperforms existing methods on challenging datasets.

Produces high-fidelity, detailed reconstructions.

Effectively models interactions and contacts between instances.

Abstract

Reconstructing dynamic scenes with multiple interacting humans and objects from sparse-view inputs is a critical yet challenging task, essential for creating high-fidelity digital twins for robotics and VR/AR. This problem, which we term Multi-Human Multi-Object (MHMO) rendering, presents two significant obstacles: achieving view-consistent representations for individual instances under severe mutual occlusion, and explicitly modeling the complex and combinatorial dependencies that arise from their interactions. To overcome these challenges, we propose MM-GS, a novel hierarchical framework built upon 3D Gaussian Splatting. Our method first employs a Per-Instance Multi-View Fusion module to establish a robust and consistent representation for each instance by aggregating visual information across all available views. Subsequently, a Scene-Level Instance Interaction module operates on a global scene graph to reason about relationships between all participants, refining their attributes to capture subtle interaction effects. Extensive experiments on challenging datasets demonstrate that our method significantly outperforms strong baselines, producing state-of-the-art results with high-fidelity details and plausible inter-instance contacts.