Q-Doc: Benchmarking Document Image Quality Assessment Capabilities in Multi-modal Large Language Models

TL;DR

This paper introduces Q-Doc, a comprehensive benchmarking framework for evaluating document image quality assessment capabilities of multi-modal large language models across multiple levels of analysis.

Contribution

It presents a novel three-tiered evaluation framework and demonstrates the current limitations and potential improvements of MLLMs in DIQA tasks.

Findings

MLLMs show limited DIQA abilities with inconsistent scoring.

Chain-of-Thought prompting improves DIQA performance.

Significant gaps remain in MLLMs' understanding of document image quality.

Abstract

The rapid advancement of Multi-modal Large Language Models (MLLMs) has expanded their capabilities beyond high-level vision tasks. Nevertheless, their potential for Document Image Quality Assessment (DIQA) remains underexplored. To bridge this gap, we propose Q-Doc, a three-tiered evaluation framework for systematically probing DIQA capabilities of MLLMs at coarse, middle, and fine granularity levels. a) At the coarse level, we instruct MLLMs to assign quality scores to document images and analyze their correlation with Quality Annotations. b) At the middle level, we design distortion-type identification tasks, including single-choice and multi-choice tests for multi-distortion scenarios. c) At the fine level, we introduce distortion-severity assessment where MLLMs classify distortion intensity against human-annotated references. Our evaluation demonstrates that while MLLMs possess nascent DIQA abilities, they exhibit critical limitations: inconsistent scoring, distortion misidentification, and severity misjudgment. Significantly, we show that Chain-of-Thought (CoT) prompting substantially enhances performance across all levels. Our work provides a benchmark for DIQA capabilities in MLLMs, revealing pronounced deficiencies in their quality perception and promising pathways for enhancement. The benchmark and code are publicly available at: https://github.com/cydxf/Q-Doc.