HSD: Training-Free Acceleration for Document Parsing Vision-Language Model with Hierarchical Speculative Decoding

TL;DR

HSD is a hierarchical decoding framework that accelerates document parsing in vision-language models by predicting regions and verifying drafts in a two-stage process, significantly reducing inference time.

Contribution

The paper introduces Hierarchical Speculative Decoding (HSD), a novel two-stage local-to-global approach that improves inference speed while maintaining global coherence in document parsing.

Findings

Achieves up to 7.04x speedup on long-document parsing tasks.

Provides near-lossless 2.78x speedup with HunyuanOCR on OmniDocBench v1.5.

Effectively preserves full-page coherence through hierarchical verification.

Abstract

Document parsing is a fundamental task in multimodal understanding, supporting a wide range of downstream applications such as information extraction and intelligent document analysis. Benefiting from strong semantic modeling and robust generalization, VLM-based end-to-end approaches have emerged as the mainstream paradigm in recent years. However, these models often suffer from substantial inference latency, as they must autoregressively generate long, full-page sequences when processing long-form documents. While recent hybrid methods mitigate this issue via region-level parallel decoding with VLMs, independent region decoding loses full-page context and might weaken global coherence. To address this issue, we propose Hierarchical Speculative Decoding (HSD), a two-stage local-to-global framework for document parsing. HSD first employs a lightweight pipeline drafter to predict region partitions and generate coarse drafts for each region. The first stage verifies the generated region-level drafts in parallel for efficiency, while the second stage further performs page-level verification on these refined outputs to preserve full-page coherence. Experimental results show that our HSD achieves a 2.78x near-lossless speedup with HunyuanOCR on OmniDocBench v1.5 and up to 7.04x speedup on long-document parsing tasks, demonstrating the effectiveness of our proposed method. We will release our code to facilitate reproducibility.