Structsum Generation for Faster Text Comprehension

TL;DR

This paper develops prompting strategies for large language models to generate structured text representations like tables and mind maps, significantly improving accuracy and reducing comprehension time in text understanding tasks.

Contribution

It introduces a taxonomy of problems, critique methods, and the Auto-QA evaluation framework for structured text generation with LLMs, achieving notable accuracy improvements.

Findings

Accuracy improved by +37pp for mind maps and +15pp for tables.

Structured representations reduce comprehension time by 42.9% (tables) and 31.9% (mind maps).

Auto-QA effectively evaluates semantic coverage of generated structures.

Abstract

We consider the task of generating structured representations of text using large language models (LLMs). We focus on tables and mind maps as representative modalities. Tables are more organized way of representing data, while mind maps provide a visually dynamic and flexible approach, particularly suitable for sparse content. Despite the effectiveness of LLMs on different tasks, we show that current models struggle with generating structured outputs. In response, we present effective prompting strategies for both of these tasks. We introduce a taxonomy of problems around factuality, global and local structure, common to both modalities and propose a set of critiques to tackle these issues resulting in an absolute improvement in accuracy of +37pp (79%) for mind maps and +15pp (78%) for tables. To evaluate semantic coverage of generated structured representations we propose Auto-QA, and we verify the adequacy of Auto-QA using SQuAD dataset. We further evaluate the usefulness of structured representations via a text comprehension user study. The results show a significant reduction in comprehension time compared to text when using table (42.9%) and mind map (31.9%), without loss in accuracy.