RoDLA: Benchmarking the Robustness of Document Layout Analysis Models

TL;DR

This paper introduces a comprehensive robustness benchmark for Document Layout Analysis models, including a large dataset, perturbation taxonomy, and evaluation metrics, and proposes RoDLA, a new model that achieves state-of-the-art robustness and accuracy.

Contribution

It is the first to establish a robustness benchmark for DLA, including datasets, perturbation taxonomy, metrics, and a new robust model, RoDLA.

Findings

RoDLA achieves state-of-the-art robustness scores on benchmark datasets.

RoDLA outperforms previous methods in mAP by significant margins.

The benchmark provides a comprehensive evaluation framework for DLA robustness.

Abstract

Before developing a Document Layout Analysis (DLA) model in real-world applications, conducting comprehensive robustness testing is essential. However, the robustness of DLA models remains underexplored in the literature. To address this, we are the first to introduce a robustness benchmark for DLA models, which includes 450K document images of three datasets. To cover realistic corruptions, we propose a perturbation taxonomy with 36 common document perturbations inspired by real-world document processing. Additionally, to better understand document perturbation impacts, we propose two metrics, Mean Perturbation Effect (mPE) for perturbation assessment and Mean Robustness Degradation (mRD) for robustness evaluation. Furthermore, we introduce a self-titled model, i.e., Robust Document Layout Analyzer (RoDLA), which improves attention mechanisms to boost extraction of robust features. Experiments on the proposed benchmarks (PubLayNet-P, DocLayNet-P, and M$^6$Doc-P) demonstrate that RoDLA obtains state-of-the-art mRD scores of 115.7, 135.4, and 150.4, respectively. Compared to previous methods, RoDLA achieves notable improvements in mAP of +3.8%, +7.1% and +12.1%, respectively.