Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs

TL;DR

This paper introduces a benchmarking framework for evaluating PDF document parsers on mathematical formula extraction, using synthetic PDFs with LaTeX ground truth and LLM-based semantic evaluation.

Contribution

It presents a novel benchmarking approach with a synthetic dataset, LLM-based semantic evaluation, and a comprehensive comparison of over 20 parsers.

Findings

LLM-based evaluation correlates highly with human judgment (r=0.78).

Significant performance disparities exist among different PDF parsers.

The framework provides actionable insights for parser selection in scientific applications.

Abstract

Correctly parsing mathematical formulas from PDFs is critical for training large language models and building scientific knowledge bases from academic literature, yet existing benchmarks either exclude formulas entirely or lack semantically-aware evaluation metrics. We introduce a benchmarking framework centered on synthetically generated PDFs with precise LaTeX ground truth, enabling systematic control over layout, formulas, and content characteristics. For evaluation, we apply LLM-as-a-judge to assess semantic equivalence of parsed formulas, capturing mathematical meaning beyond surface-level notation differences. We validate this approach through a human study (250 formula pairs, 750 ratings from 30 evaluators), showing a Pearson correlation of r=0.78 with human judgment, compared to r=0.34 for character-level matching (CDM) and r~0 for text similarity. Our robust two-stage matching pipeline combining LLM-based extraction with fuzzy validation reliably aligns parsed formulas with ground truth despite format inconsistencies across parsers. Evaluating 20+ contemporary PDF parsers across 100 synthetic documents with 2,000+ formulas reveals significant performance disparities, providing actionable guidance for practitioners selecting parsers for downstream applications. Code and benchmark data: https://github.com/phorn1/pdf-parse-bench and https://github.com/phorn1/formula-metric-study