InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model

TL;DR

InstaInpaint is a fast, reference-based 3D scene inpainting framework that produces high-quality results in under half a second, enabling real-time editing and object manipulation in virtual environments.

Contribution

The paper introduces a novel large reconstruction model trained with self-supervised masked-finetuning, achieving real-time 3D scene inpainting with significantly improved speed and comparable quality.

Findings

Achieves 1000x speed-up over previous methods.

Maintains state-of-the-art performance on standard benchmarks.

Generalizes well to object insertion and multi-region inpainting.

Abstract

Recent advances in 3D scene reconstruction enable real-time viewing in virtual and augmented reality. To support interactive operations for better immersiveness, such as moving or editing objects, 3D scene inpainting methods are proposed to repair or complete the altered geometry. However, current approaches rely on lengthy and computationally intensive optimization, making them impractical for real-time or online applications. We propose InstaInpaint, a reference-based feed-forward framework that produces 3D-scene inpainting from a 2D inpainting proposal within 0.4 seconds. We develop a self-supervised masked-finetuning strategy to enable training of our custom large reconstruction model (LRM) on the large-scale dataset. Through extensive experiments, we analyze and identify several key designs that improve generalization, textural consistency, and geometric correctness. InstaInpaint achieves a 1000x speed-up from prior methods while maintaining a state-of-the-art performance across two standard benchmarks. Moreover, we show that InstaInpaint generalizes well to flexible downstream applications such as object insertion and multi-region inpainting. More video results are available at our project page: https://dhmbb2.github.io/InstaInpaint_page/.