CObL: Toward Zero-Shot Ordinal Layering without User Prompting

TL;DR

CObL introduces a diffusion-based model that infers occlusion-ordered object layers from images, enabling zero-shot generalization to real-world scenes without user prompting or prior object count knowledge.

Contribution

It presents a novel zero-shot method for reconstructing multiple occluded objects in scenes without user prompts or prior object count, using diffusion models and synthetic training.

Findings

Zero-shot generalization to real-world scenes

Reconstructs multiple occluded objects without prompts

Does not require prior knowledge of object count

Abstract

Vision benefits from grouping pixels into objects and understanding their spatial relationships, both laterally and in depth. We capture this with a scene representation comprising an occlusion-ordered stack of "object layers," each containing an isolated and amodally-completed object. To infer this representation from an image, we introduce a diffusion-based architecture named Concurrent Object Layers (CObL). CObL generates a stack of object layers in parallel, using Stable Diffusion as a prior for natural objects and inference-time guidance to ensure the inferred layers composite back to the input image. We train CObL using a few thousand synthetically-generated images of multi-object tabletop scenes, and we find that it zero-shot generalizes to photographs of real-world tabletops with varying numbers of novel objects. In contrast to recent models for amodal object completion, CObL reconstructs multiple occluded objects without user prompting and without knowing the number of objects beforehand. Unlike previous models for unsupervised object-centric representation learning, CObL is not limited to the world it was trained in.