Uncertainty-Aware Complex Scientific Table Data Extraction

TL;DR

This paper introduces an uncertainty-aware framework for extracting data from complex scientific tables, leveraging conformal prediction to improve accuracy and reduce manual verification efforts.

Contribution

It presents a novel uncertainty quantification method for scientific table data extraction, enhancing error detection and reducing manual verification workload.

Findings

Using UQ, data quality improves by 30% with 47% manual verification.

Framework effectively detects extraction errors in complex scientific tables.

Quantitative evaluation demonstrates the potential of UQ to streamline data extraction workflows.

Abstract

Table structure recognition (TSR) and optical character recognition (OCR) play crucial roles in extracting structured data from tables in scientific documents. However, existing extraction frameworks built on top of TSR and OCR methods often fail to quantify the uncertainties of extracted results. To obtain highly accurate data for scientific domains, all extracted data must be manually verified, which can be time-consuming and labor-intensive. We propose a framework that performs uncertainty-aware data extraction for complex scientific tables, built on conformal prediction, a model-agnostic method for uncertainty quantification (UQ). We explored various uncertainty scoring methods to aggregate the uncertainties introduced by TSR and OCR. We rigorously evaluated the framework using a standard benchmark and an in-house dataset consisting of complex scientific tables in six scientific domains. The results demonstrate the effectiveness of using UQ for extraction error detection, and by manually verifying only 47% of extraction results, the data quality can be improved by 30%. Our work quantitatively demonstrates the role of UQ with the potential of improving the efficiency in the human-machine cooperation process to obtain scientifically usable data from complex tables in scientific documents. All code and data are available on GitHub at https://github.com/lamps-lab/TSR-OCR-UQ/tree/main.