Geometry Aligned Variational Transformer for Image-conditioned Layout Generation

TL;DR

This paper introduces ICVT, a novel transformer-based model that generates image-conditioned layouts by integrating visual and geometric information, improving aesthetic and semantic coherence in layout design.

Contribution

The paper proposes a new image-conditioned layout generation paradigm with a geometry alignment module and a variational transformer, enabling diverse and harmonious layout creation.

Findings

ICVT effectively models intra-layout relationships and fuses visual information.

The geometry alignment module improves the coherence between layout and image geometry.

Experimental results demonstrate superior layout quality and diversity.

Abstract

Layout generation is a novel task in computer vision, which combines the challenges in both object localization and aesthetic appraisal, widely used in advertisements, posters, and slides design. An accurate and pleasant layout should consider both the intra-domain relationship within layout elements and the inter-domain relationship between layout elements and the image. However, most previous methods simply focus on image-content-agnostic layout generation, without leveraging the complex visual information from the image. To this end, we explore a novel paradigm entitled image-conditioned layout generation, which aims to add text overlays to an image in a semantically coherent manner. Specifically, we propose an Image-Conditioned Variational Transformer (ICVT) that autoregressively generates various layouts in an image. First, self-attention mechanism is adopted to model the contextual relationship within layout elements, while cross-attention mechanism is used to fuse the visual information of conditional images. Subsequently, we take them as building blocks of conditional variational autoencoder (CVAE), which demonstrates appealing diversity. Second, in order to alleviate the gap between layout elements domain and visual domain, we design a Geometry Alignment module, in which the geometric information of the image is aligned with the layout representation. In addition, we construct a large-scale advertisement poster layout designing dataset with delicate layout and saliency map annotations. Experimental results show that our model can adaptively generate layouts in the non-intrusive area of the image, resulting in a harmonious layout design.