Towards Improving the Generation Quality of Autoregressive Slot VAEs

TL;DR

This paper enhances autoregressive slot VAEs by conditioning on scene-level variables and learning a consistent object order, significantly improving unconditional scene generation quality.

Contribution

It introduces two novel improvements: scene-level conditioning and learned object ordering, to better model object correlations in scene generation.

Findings

Improved unconditional scene generation quality across three environments.

Validated effectiveness of scene-level conditioning and object ordering through ablation studies.

Achieved significant gains over baseline models in generating coherent multi-object scenes.

Abstract

Unconditional scene inference and generation are challenging to learn jointly with a single compositional model. Despite encouraging progress on models that extract object-centric representations (''slots'') from images, unconditional generation of scenes from slots has received less attention. This is primarily because learning the multi-object relations necessary to imagine coherent scenes is difficult. We hypothesize that most existing slot-based models have a limited ability to learn object correlations. We propose two improvements that strengthen object correlation learning. The first is to condition the slots on a global, scene-level variable that captures higher-order correlations between slots. Second, we address the fundamental lack of a canonical order for objects in images by proposing to learn a consistent order to use for the autoregressive generation of scene objects. Specifically, we train an autoregressive slot prior to sequentially generate scene objects following a learned order. Ordered slot inference entails first estimating a randomly ordered set of slots using existing approaches for extracting slots from images, then aligning those slots to ordered slots generated autoregressively with the slot prior. Our experiments across three multi-object environments demonstrate clear gains in unconditional scene generation quality. Detailed ablation studies are also provided that validate the two proposed improvements.