Object-level Scene Deocclusion

TL;DR

This paper introduces PACO, a self-supervised diffusion framework that effectively deoccludes objects in scenes by predicting full-view features from partial views, leveraging large-scale training and pre-trained models.

Contribution

PACO is a novel self-supervised foundation model for object-level scene deocclusion, combining parallel autoencoders and latent prediction to handle occlusions without manual annotations.

Findings

PACO surpasses state-of-the-art methods on COCOA and real-world scenes.

It generalizes well to cross-domain scenes and unseen categories.

The model enables applications in 3D reconstruction and object recomposition.

Abstract

Deoccluding the hidden portions of objects in a scene is a formidable task, particularly when addressing real-world scenes. In this paper, we present a new self-supervised PArallel visible-to-COmplete diffusion framework, named PACO, a foundation model for object-level scene deocclusion. Leveraging the rich prior of pre-trained models, we first design the parallel variational autoencoder, which produces a full-view feature map that simultaneously encodes multiple complete objects, and the visible-to-complete latent generator, which learns to implicitly predict the full-view feature map from partial-view feature map and text prompts extracted from the incomplete objects in the input image. To train PACO, we create a large-scale dataset with 500k samples to enable self-supervised learning, avoiding tedious annotations of the amodal masks and occluded regions. At inference, we devise a layer-wise deocclusion strategy to improve efficiency while maintaining the deocclusion quality. Extensive experiments on COCOA and various real-world scenes demonstrate the superior capability of PACO for scene deocclusion, surpassing the state of the arts by a large margin. Our method can also be extended to cross-domain scenes and novel categories that are not covered by the training set. Further, we demonstrate the deocclusion applicability of PACO in single-view 3D scene reconstruction and object recomposition.