Seeing Through Clutter: Structured 3D Scene Reconstruction via Iterative Object Removal

TL;DR

SeeingThroughClutter introduces an iterative object removal approach for 3D scene reconstruction from single images, improving robustness in cluttered scenes without task-specific training by leveraging foundation models.

Contribution

The paper proposes a novel iterative object removal pipeline that enhances 3D scene reconstruction in complex scenes without requiring task-specific training.

Findings

Achieves state-of-the-art robustness on 3D-Front and ADE20K datasets.

Effectively handles occlusion and clutter through iterative object removal.

Utilizes foundation models to orchestrate the process without additional training.

Abstract

We present SeeingThroughClutter, a method for reconstructing structured 3D representations from single images by segmenting and modeling objects individually. Prior approaches rely on intermediate tasks such as semantic segmentation and depth estimation, which often underperform in complex scenes, particularly in the presence of occlusion and clutter. We address this by introducing an iterative object removal and reconstruction pipeline that decomposes complex scenes into a sequence of simpler subtasks. Using VLMs as orchestrators, foreground objects are removed one at a time via detection, segmentation, object removal, and 3D fitting. We show that removing objects allows for cleaner segmentations of subsequent objects, even in highly occluded scenes. Our method requires no task-specific training and benefits directly from ongoing advances in foundation models. We demonstrate stateof-the-art robustness on 3D-Front and ADE20K datasets. Project Page: https://rioak.github.io/seeingthroughclutter/