MathDoc: Benchmarking Structured Extraction and Active Refusal on Noisy Mathematics Exam Papers

TL;DR

MathDoc is a new benchmark for evaluating the ability of models to extract structured information from noisy, real-world mathematics exam papers and to actively refuse illegible inputs, highlighting a critical gap in current models' reliability.

Contribution

We introduce MathDoc, the first benchmark focusing on document-level extraction and refusal behavior on authentic noisy math exam papers, with a comprehensive evaluation framework.

Findings

State-of-the-art models struggle to refuse illegible inputs.

Models produce confident but invalid outputs on noisy data.

MathDoc reveals a significant gap in model reliability under degraded conditions.

Abstract

The automated extraction of structured questions from paper-based mathematics exams is fundamental to intelligent education, yet remains challenging in real-world settings due to severe visual noise. Existing benchmarks mainly focus on clean documents or generic layout analysis, overlooking both the structural integrity of mathematical problems and the ability of models to actively reject incomplete inputs. We introduce MathDoc, the first benchmark for document-level information extraction from authentic high school mathematics exam papers. MathDoc contains \textbf{3,609} carefully curated questions with real-world artifacts and explicitly includes unrecognizable samples to evaluate active refusal behavior. We propose a multi-dimensional evaluation framework covering stem accuracy, visual similarity, and refusal capability. Experiments on SOTA MLLMs, including Qwen3-VL and Gemini-2.5-Pro, show that although end-to-end models achieve strong extraction performance, they consistently fail to refuse illegible inputs, instead producing confident but invalid outputs. These results highlight a critical gap in current MLLMs and establish MathDoc as a benchmark for assessing model reliability under degraded document conditions. Our project repository is available at \href{https://github.com/winnk123/papers/tree/master}{GitHub repository}