DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems

TL;DR

DocBench is a comprehensive benchmark designed to evaluate the performance of LLM-based document reading systems across diverse real-world scenarios, highlighting current gaps and guiding future improvements.

Contribution

This paper introduces DocBench, the first standardized benchmark for assessing LLM-based document reading systems with real documents and synthetic questions across multiple domains.

Findings

Existing systems lag behind human performance.

Significant challenges remain in multi-modal and long-context understanding.

Benchmark reveals gaps and guides future research.

Abstract

Recently, there has been a growing interest among large language model (LLM) developers in LLM-based document reading systems, which enable users to upload their own documents and pose questions related to the document contents, going beyond simple reading comprehension tasks. Consequently, these systems have been carefully designed to tackle challenges such as file parsing, metadata extraction, multi-modal information understanding and long-context reading. However, no current benchmark exists to evaluate their performance in such scenarios, where a raw file and questions are provided as input, and a corresponding response is expected as output. In this paper, we introduce DocBench, a new benchmark designed to evaluate LLM-based document reading systems. Our benchmark involves a meticulously crafted process, including the recruitment of human annotators and the generation of synthetic questions. It includes 229 real documents and 1,102 questions, spanning across five different domains and four major types of questions. We evaluate both proprietary LLM-based systems accessible via web interfaces or APIs, and a parse-then-read pipeline employing open-source LLMs. Our evaluations reveal noticeable gaps between existing LLM-based document reading systems and human performance, underscoring the challenges of developing proficient systems. To summarize, DocBench aims to establish a standardized benchmark for evaluating LLM-based document reading systems under diverse real-world scenarios, thereby guiding future advancements in this research area.