Rein3D: Reinforced 3D Indoor Scene Generation with Panoramic Video Diffusion Models

TL;DR

Rein3D introduces a novel framework combining 3D Gaussian Splatting with video diffusion models to synthesize high-quality, globally consistent 3D indoor scenes from sparse inputs, addressing missing geometry and occlusions.

Contribution

The paper presents Rein3D, a new approach that integrates panoramic video diffusion and 3D Gaussian Splatting for improved 3D scene reconstruction and restoration.

Findings

Rein3D generates photorealistic, globally consistent 3D indoor scenes.

The method significantly enhances long-range camera exploration.

Rein3D outperforms existing baselines in scene synthesis quality.

Abstract

The growing demand for Embodied AI and VR applications has highlighted the need for synthesizing high-quality 3D indoor scenes from sparse inputs. However, existing approaches struggle to infer massive amounts of missing geometry in large unseen areas while maintaining global consistency, often producing locally plausible but globally inconsistent reconstructions. We present Rein3D, a framework that reconstructs full 360-degree indoor environments by coupling explicit 3D Gaussian Splatting (3DGS) with temporally coherent priors from video diffusion models. Our approach follows a "restore-and-refine" paradigm: we employ a radial exploration strategy to render imperfect panoramic videos along trajectories starting from the origin, effectively uncovering occluded regions from a coarse 3DGS initialization. These sequences are restored by a panoramic video-to-video diffusion model and further enhanced via video super-resolution to synthesize high-fidelity geometry and textures. Finally, these refined videos serve as pseudo-ground truths to update the global 3D Gaussian field. To support this task, we construct PanoV2V-15K, a dataset of over 15K paired clean and degraded panoramic videos for diffusion-based scene restoration. Experiments demonstrate that Rein3D produces photorealistic and globally consistent 3D scenes and significantly improves long-range camera exploration compared with existing baselines.