Internalized Reasoning for Long-Context Visual Document Understanding

TL;DR

This paper presents a synthetic data pipeline for enhancing reasoning in long-document visual understanding, leading to improved performance and efficiency in enterprise, legal, and scientific tasks.

Contribution

It introduces a novel synthetic reasoning data pipeline and internalized reasoning method, significantly boosting performance on long-document benchmarks.

Findings

Achieved 58.3 on MMLongBenchDoc with Qwen3 VL, surpassing larger models.

Synthetic reasoning outperforms distillation by 3.8 points on MMLBD-C.

Internalized reasoning reduces output tokens by 12.4 times.

Abstract

Visual long-document understanding is critical for enterprise, legal, and scientific applications, yet the best performing open recipes have not explored reasoning, a capability which has driven leaps in math and code performance. We introduce a synthetic data pipeline for reasoning in long-document understanding that generates thinking traces by scoring each page for question relevance, extracting textual evidence and ordering it from most to least relevant. We apply SFT to the resulting traces within \texttt{<think>} tags, gated by a \texttt{<cot>} control token, and the resulting reasoning capability is internalized via low-strength model merging. We study Qwen3 VL 32B and Mistral Small 3.1 24B. With Qwen3 VL, we achieve 58.3 on MMLongBenchDoc, surpassing the 7$\times$ larger Qwen3 VL 235B A22B (57.0). With Mistral, we show that synthetic reasoning outperforms distillation from the Thinking version's traces by 3.8 points on MMLBD-C, and internalized reasoning exhibits 12.4$\times$ fewer mean output tokens compared to explicit reasoning. We release our pipeline for reproducibility and further exploration.