AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine

TL;DR

AdaParse is an adaptive, resource-efficient PDF parsing engine that intelligently assigns parsers to scientific documents, significantly boosting throughput while maintaining high accuracy for large-scale scientific text processing.

Contribution

It introduces a data-driven, adaptive approach that optimally selects parsers based on document complexity and human preferences, enhancing scalability and accuracy in scientific PDF parsing.

Findings

Achieves 17x throughput improvement over state-of-the-art parsers.

Maintains or slightly improves accuracy (by 0.2%) on a benchmark of 1000 documents.

Enables large-scale scientific document corpus processing for high-quality dataset creation.

Abstract

Language models for scientific tasks are trained on text from scientific publications, most distributed as PDFs that require parsing. PDF parsing approaches range from inexpensive heuristics (for simple documents) to computationally intensive ML-driven systems (for complex or degraded ones). The choice of the "best" parser for a particular document depends on its computational cost and the accuracy of its output. To address these issues, we introduce an Adaptive Parallel PDF Parsing and Resource Scaling Engine (AdaParse), a data-driven strategy for assigning an appropriate parser to each document. We enlist scientists to select preferred parser outputs and incorporate this information through direct preference optimization (DPO) into AdaParse, thereby aligning its selection process with human judgment. AdaParse then incorporates hardware requirements and predicted accuracy of each parser to orchestrate computational resources efficiently for large-scale parsing campaigns. We demonstrate that AdaParse, when compared to state-of-the-art parsers, improves throughput by $17\times$ while still achieving comparable accuracy (0.2 percent better) on a benchmark set of 1000 scientific documents. AdaParse's combination of high accuracy and parallel scalability makes it feasible to parse large-scale scientific document corpora to support the development of high-quality, trillion-token-scale text datasets. The implementation is available at https://github.com/7shoe/AdaParse/