Robustness Evaluation of OCR-based Visual Document Understanding under Multi-Modal Adversarial Attacks

TL;DR

This paper presents a comprehensive framework for generating multi-modal adversarial attacks on OCR-based Visual Document Understanding systems, revealing their vulnerabilities and guiding robustness improvements.

Contribution

It introduces the first unified method for multi-modal adversarial attacks on VDU models, covering layout, pixel, and text manipulations with realistic constraints.

Findings

Line-level attacks cause significant performance drops.

Compound perturbations are more effective than single-modality attacks.

PGD-based layout attacks outperform random-shift baselines.

Abstract

Visual Document Understanding (VDU) systems have achieved strong performance in information extraction by integrating textual, layout, and visual signals. However, their robustness under realistic adversarial perturbations remains insufficiently explored. We introduce the first unified framework for generating and evaluating multi-modal adversarial attacks on OCR-based VDU models. Our method covers six gradient-based layout attack scenarios, incorporating manipulations of OCR bounding boxes, pixels, and texts across both word and line granularities, with constraints on layout perturbation budget (e.g., IoU >= 0.6) to preserve plausibility. Experimental results across four datasets (FUNSD, CORD, SROIE, DocVQA) and six model families demonstrate that line-level attacks and compound perturbations (BBox + Pixel + Text) yield the most severe performance degradation. Projected Gradient Descent (PGD)-based BBox perturbations outperform random-shift baselines in all investigated models. Ablation studies further validate the impact of layout budget, text modification, and adversarial transferability.