Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation

TL;DR

This paper introduces an innovative approach combining graph networks and large language models to generate human-centric layouts efficiently, preserving structural integrity and enabling editable, diverse designs for mobile applications.

Contribution

The paper presents the Aggregation Structural Representation (ASR) module that integrates graph features with LLMs, replacing traditional vision modules for improved layout generation.

Findings

ASR outperforms existing methods on the RICO dataset in mIoU.

The approach enables human-editable, progressive layout design.

Sampling relational features yields diverse and creative layouts.

Abstract

Time consumption and the complexity of manual layout design make automated layout generation a critical task, especially for multiple applications across different mobile devices. Existing graph-based layout generation approaches suffer from limited generative capability, often resulting in unreasonable and incompatible outputs. Meanwhile, vision based generative models tend to overlook the original structural information, leading to component intersections and overlaps. To address these challenges, we propose an Aggregation Structural Representation (ASR) module that integrates graph networks with large language models (LLMs) to preserve structural information while enhancing generative capability. This novel pipeline utilizes graph features as hierarchical prior knowledge, replacing the traditional Vision Transformer (ViT) module in multimodal large language models (MLLM) to predict full layout information for the first time. Moreover, the intermediate graph matrix used as input for the LLM is human editable, enabling progressive, human centric design generation. A comprehensive evaluation on the RICO dataset demonstrates the strong performance of ASR, both quantitatively using mean Intersection over Union (mIoU), and qualitatively through a crowdsourced user study. Additionally, sampling on relational features ensures diverse layout generation, further enhancing the adaptability and creativity of the proposed approach.