Bi-VLDoc: Bidirectional Vision-Language Modeling for Visually-Rich Document Understanding

TL;DR

Bi-VLDoc introduces a bidirectional vision-language pre-training paradigm with hybrid-attention to improve the understanding of visually-rich documents, significantly enhancing performance across multiple document understanding benchmarks.

Contribution

It proposes a novel bidirectional supervision strategy and hybrid-attention mechanism for better cross-modal representation learning in document understanding.

Findings

Achieved state-of-the-art results on Form Understanding, Receipt Information Extraction, and Document Classification.

Significantly improved performance on Document Visual QA.

Demonstrated the effectiveness of bidirectional supervision and hybrid-attention in multi-modal document modeling.

Abstract

Multi-modal document pre-trained models have proven to be very effective in a variety of visually-rich document understanding (VrDU) tasks. Though existing document pre-trained models have achieved excellent performance on standard benchmarks for VrDU, the way they model and exploit the interactions between vision and language on documents has hindered them from better generalization ability and higher accuracy. In this work, we investigate the problem of vision-language joint representation learning for VrDU mainly from the perspective of supervisory signals. Specifically, a pre-training paradigm called Bi-VLDoc is proposed, in which a bidirectional vision-language supervision strategy and a vision-language hybrid-attention mechanism are devised to fully explore and utilize the interactions between these two modalities, to learn stronger cross-modal document representations with richer semantics. Benefiting from the learned informative cross-modal document representations, Bi-VLDoc significantly advances the state-of-the-art performance on three widely-used document understanding benchmarks, including Form Understanding (from 85.14% to 93.44%), Receipt Information Extraction (from 96.01% to 97.84%), and Document Classification (from 96.08% to 97.12%). On Document Visual QA, Bi-VLDoc achieves the state-of-the-art performance compared to previous single model methods.