Illiterate DALL-E Learns to Compose

TL;DR

This paper introduces SLATE, a novel slot-based autoencoding model that learns object-centric representations enabling zero-shot image generation without text prompts, combining systematic generalization with simplicity.

Contribution

SLATE is a new architecture that merges object-centric learning with zero-shot image generation, eliminating the need for text prompts and using a novel decoder conditioned on slots.

Findings

Significant improvement in zero-shot image generation quality.

Qualitatively comparable or better slot-attention structures.

Effective in both in-distribution and out-of-distribution scenarios.

Abstract

Although DALL-E has shown an impressive ability of composition-based systematic generalization in image generation, it requires the dataset of text-image pairs and the compositionality is provided by the text. In contrast, object-centric representation models like the Slot Attention model learn composable representations without the text prompt. However, unlike DALL-E its ability to systematically generalize for zero-shot generation is significantly limited. In this paper, we propose a simple but novel slot-based autoencoding architecture, called SLATE, for combining the best of both worlds: learning object-centric representations that allows systematic generalization in zero-shot image generation without text. As such, this model can also be seen as an illiterate DALL-E model. Unlike the pixel-mixture decoders of existing object-centric representation models, we propose to use the Image GPT decoder conditioned on the slots for capturing complex interactions among the slots and pixels. In experiments, we show that this simple and easy-to-implement architecture not requiring a text prompt achieves significant improvement in in-distribution and out-of-distribution (zero-shot) image generation and qualitatively comparable or better slot-attention structure than the models based on mixture decoders.