World Reconstruction From Inconsistent Views

TL;DR

This paper introduces a method to reconstruct consistent 3D worlds from videos with inconsistent frames by aligning and optimizing pointclouds, enabling high-quality 3D environment generation from video diffusion models.

Contribution

The paper presents a novel non-rigid alignment and global optimization approach for 3D reconstruction from inconsistent video frames, improving 3D scene quality and consistency.

Findings

Higher quality 3D reconstructions than baselines

Effective alignment of inconsistent video frames

Generation of explorable 3D environments

Abstract

Video diffusion models generate high-quality and diverse worlds; however, individual frames often lack 3D consistency across the output sequence, which makes the reconstruction of 3D worlds difficult. To this end, we propose a new method that handles these inconsistencies by non-rigidly aligning the video frames into a globally-consistent coordinate frame that produces sharp and detailed pointcloud reconstructions. First, a geometric foundation model lifts each frame into a pixel-wise 3D pointcloud, which contains unaligned surfaces due to these inconsistencies. We then propose a tailored non-rigid iterative frame-to-model ICP to obtain an initial alignment across all frames, followed by a global optimization that further sharpens the pointcloud. Finally, we leverage this pointcloud as initialization for 3D reconstruction and propose a novel inverse deformation rendering loss to create high quality and explorable 3D environments from inconsistent views. We demonstrate that our 3D scenes achieve higher quality than baselines, effectively turning video models into 3D-consistent world generators.