From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering

TL;DR

This study systematically evaluates how different PDF processing frameworks and configurations affect the accuracy of domain-specific question answering in RAG systems, highlighting data preparation as a key factor.

Contribution

It provides the first comprehensive comparison of open-source PDF-to-Markdown conversion tools and their impact on RAG question-answering accuracy.

Findings

Docling with hierarchical splitting achieved 94.1% accuracy.

Metadata enrichment and hierarchy-aware chunking significantly improved results.

Font-based hierarchy rebuilding outperformed LLM-based approaches.

Abstract

Retrieval-Augmented Generation (RAG) systems depend critically on the quality of document preprocessing, yet no prior study has evaluated PDF processing frameworks by their impact on downstream question-answering accuracy. We address this gap through a systematic comparison of four open-source PDF-to-Markdown conversion frameworks, Docling, MinerU, Marker, and DeepSeek OCR, across 19 pipeline configurations for extracting text and other contents from PDFs, varying the conversion tool, cleaning transformations, splitting strategy, and metadata enrichment. Evaluation was performed using a manually curated 50-question benchmark over a corpus of 36 Portuguese administrative documents (1,706 pages, ~492K words), with LLM-as-judge scoring averaged over 10 runs. Two baselines bounded the results: na\"ive PDFLoader (86.9%) and manually curated Markdown (97.1%). Docling with hierarchical splitting and image descriptions achieved the highest automated accuracy (94.1%). Metadata enrichment and hierarchy-aware chunking contributed more to accuracy than the conversion framework choice alone. Font-based hierarchy rebuilding consistently outperformed LLM-based approaches. An exploratory GraphRAG implementation scored only 82%, underperforming basic RAG, suggesting that na\"ive knowledge graph construction without ontological guidance does not yet justify its added complexity. These findings demonstrate that data preparation quality is the dominant factor in RAG system performance.