Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding

TL;DR

This paper introduces a method to improve the robustness of vision-language models in understanding complex document layouts by pre-resolving layout entities with a lightweight detector, significantly enhancing out-of-distribution performance.

Contribution

The authors propose a novel layout prior injection technique that shares the decoder's generation space, reducing failures without altering the base model architecture.

Findings

Markdown F1 improved from 0.37 to 0.92 on a 10k-page benchmark.

Table TEDS increased from 0.01 to 0.36 on OmniDocBench.

Decoding failures dropped across multiple industrial domains.

Abstract

Vision-Language Models (VLMs) parse documents end-to-end but frequently break down on layouts unlike those seen in training. We attribute this to a two-hop bottleneck: before the decoder can extract content (Hop 2), it must first classify and localize the enclosing layout entity (Hop 1), and when the first hop fails the second collapses into omissions, malformed structure, or autoregressive repetition. We pre-resolve Hop 1 outside the decoder by running a lightweight RT-DETR detector, serializing its outputs in the parser's native DocTags vocabulary, and injecting them into the prompt alongside the full page image. Unlike analyze-then-parse approaches that crop the page, or prior prompt-level priors written in plain text, our prior shares the decoder's generation space and leaves the global image in view as a fallback when detections are noisy. On a 10k-page structural out-of-distribution benchmark, markdown F1 rises from $0.37$ to $0.92$; on the Chinese subset of OmniDocBench, table TEDS rises from $0.01$ to $0.36$; and on the 26k-page ViDoRe V3 benchmark, infinite-loop decoding failures drop across every industrial domain tested. These gains cost $15\%$ wall-clock latency and a median of $74$ prompt tokens, with no architectural change to the base VLM. An attention-level analysis further reveals a bimodal phase shift in which the decoder attends to injected layout tokens when emitting structure and to image patches when emitting content, consistent with the two-hop bottleneck being alleviated. Model weights will be released to support reproducibility.