Large Language Models Understand Layout

TL;DR

This paper demonstrates that large language models can understand and reason about text layouts using spatial markers, with their abilities enhanced through specific training data and instruction tuning, benefiting visual question-answering systems.

Contribution

It reveals that LLMs' layout understanding stems from pretraining data and instruction tuning, and introduces a novel auto-generated data approach to improve this ability.

Findings

LLMs can answer spatial reasoning questions with layout cues

Layout understanding improves with instruction tuning and auto-generated data

Layout understanding enhances visual question-answering performance

Abstract

Large language models (LLMs) demonstrate extraordinary abilities in a wide range of natural language processing (NLP) tasks. In this paper, we show that, beyond text understanding capability, LLMs are capable of processing text layouts that are denoted by spatial markers. They are able to answer questions that require explicit spatial perceiving and reasoning, while a drastic performance drop is observed when the spatial markers from the original data are excluded. We perform a series of experiments with the GPT-3.5, Baichuan2, Llama2 and ChatGLM3 models on various types of layout-sensitive datasets for further analysis. The experimental results reveal that the layout understanding ability of LLMs is mainly introduced by the coding data for pretraining, which is further enhanced at the instruction-tuning stage. In addition, layout understanding can be enhanced by integrating low-cost, auto-generated data approached by a novel text game. Finally, we show that layout understanding ability is beneficial for building efficient visual question-answering (VQA) systems.