A Comparative Study of PDF Parsing Tools Across Diverse Document Categories

TL;DR

This study compares 10 PDF parsing tools across six document categories, evaluating their effectiveness in text extraction and table detection, revealing strengths and weaknesses tailored to document types and tasks.

Contribution

It provides a comprehensive comparison of PDF parsing tools across diverse document types, highlighting their performance variations and guiding tool selection for specific applications.

Findings

PyMuPDF and pypdfium excel in text extraction overall.

Learning-based tools like Nougat perform better on scientific and patent documents.

TATR outperforms in table detection for scientific and financial documents.

Abstract

PDF is one of the most prominent data formats, making PDF parsing crucial for information extraction and retrieval, particularly with the rise of RAG systems. While various PDF parsing tools exist, their effectiveness across different document types remains understudied, especially beyond academic papers. Our research aims to address this gap by comparing 10 popular PDF parsing tools across 6 document categories using the DocLayNet dataset. These tools include PyPDF, pdfminer-six, PyMuPDF, pdfplumber, pypdfium2, Unstructured, Tabula, Camelot, as well as the deep learning-based tools Nougat and Table Transformer(TATR). We evaluated both text extraction and table detection capabilities. For text extraction, PyMuPDF and pypdfium generally outperformed others, but all parsers struggled with Scientific and Patent documents. For these challenging categories, learning-based tools like Nougat demonstrated superior performance. In table detection, TATR excelled in the Financial, Patent, Law & Regulations, and Scientific categories. Table detection tool Camelot performed best for tender documents, while PyMuPDF performed superior in the Manual category. Our findings highlight the importance of selecting appropriate parsing tools based on document type and specific tasks, providing valuable insights for researchers and practitioners working with diverse document sources.