Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction

TL;DR

This paper introduces the AOE benchmark to evaluate LLMs' ability to convert fragmented documents into organized tables, revealing significant challenges even for state-of-the-art models across diverse tasks.

Contribution

The paper presents a new bilingual benchmark with diverse tasks and data to systematically assess LLMs' structured table construction capabilities from complex documents.

Findings

Most advanced LLMs perform poorly on the benchmark.

The benchmark covers 11 tasks across three domains.

Models struggle with generating organized, context-specific tables.

Abstract

With the emergence of large language models (LLMs), there is an expectation that LLMs can effectively extract explicit information from complex real-world documents (e.g., papers, reports). However, most LLMs generate paragraph-style answers that are chaotic, disorganized, and untraceable. To bridge this gap, we introduce the Arranged and Organized Extraction Benchmark (AOE), a new bilingual benchmark with data and documents of varying lengths designed to systematically evaluate the ability of LLMs to comprehend fragmented documents and reconstruct isolated information into one organized table. Unlike conventional text-to-table tasks, which rely on fixed schema and narrow task domains, AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries. In the experiment, we evaluated both open-source and closed-source state-of-the-art LLMs. The results show that even the most advanced models struggled significantly. The benchmark is available at https://anonymous.4open.science/r/AOE-Benchmark/.