LayoutTransformer: Layout Generation and Completion with Self-attention

TL;DR

LayoutTransformer is a self-attention based framework that learns contextual relationships between layout elements to generate and extend diverse scene layouts across multiple domains, capturing semantic properties of primitives.

Contribution

It introduces LayoutTransformer, a novel self-attention model capable of generating and completing layouts in various domains with scalable primitive support.

Findings

Achieves competitive performance across diverse datasets.

Automatically captures semantic properties of primitives.

Supports layout generation from empty or seed sets.

Abstract

We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents, and 3D objects. Most complex scenes, natural or human-designed, can be expressed as a meaningful arrangement of simpler compositional graphical primitives. Generating a new layout or extending an existing layout requires understanding the relationships between these primitives. To do this, we propose LayoutTransformer, a novel framework that leverages self-attention to learn contextual relationships between layout elements and generate novel layouts in a given domain. Our framework allows us to generate a new layout either from an empty set or from an initial seed set of primitives, and can easily scale to support an arbitrary of primitives per layout. Furthermore, our analyses show that the model is able to automatically capture the semantic properties of the primitives. We propose simple improvements in both representation of layout primitives, as well as training methods to demonstrate competitive performance in very diverse data domains such as object bounding boxes in natural images(COCO bounding box), documents (PubLayNet), mobile applications (RICO dataset) as well as 3D shapes (Part-Net). Code and other materials will be made available at https://kampta.github.io/layout.