Compositional Scene Understanding through Inverse Generative Modeling

TL;DR

This paper introduces a compositional inverse generative modeling approach for scene understanding, enabling robust inference of scene structure and global factors from images, even with novel objects and configurations.

Contribution

It proposes a novel compositional inverse modeling framework that generalizes scene understanding to unseen objects and scenes, leveraging smaller models and pretrained generative models.

Findings

Enables inference of objects and scene factors from natural images.

Generalizes to scenes with new objects and configurations.

Applicable to pretrained text-to-image models for zero-shot perception.

Abstract

Generative models have demonstrated remarkable abilities in generating high-fidelity visual content. In this work, we explore how generative models can further be used not only to synthesize visual content but also to understand the properties of a scene given a natural image. We formulate scene understanding as an inverse generative modeling problem, where we seek to find conditional parameters of a visual generative model to best fit a given natural image. To enable this procedure to infer scene structure from images substantially different than those seen during training, we further propose to build this visual generative model compositionally from smaller models over pieces of a scene. We illustrate how this procedure enables us to infer the set of objects in a scene, enabling robust generalization to new test scenes with an increased number of objects of new shapes. We further illustrate how this enables us to infer global scene factors, likewise enabling robust generalization to new scenes. Finally, we illustrate how this approach can be directly applied to existing pretrained text-to-image generative models for zero-shot multi-object perception. Code and visualizations are at https://energy-based-model.github.io/compositional-inference.