DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents

TL;DR

DocumentCLIP introduces a contrastive learning framework that enhances vision-language models to understand complex intra-document relationships between images and text, improving performance on multimodal document understanding tasks.

Contribution

It is the first to explore multimodal intra-document links using contrastive learning and leverages a large Wikipedia dataset for pretraining.

Findings

Outperforms state-of-the-art baselines in supervised tasks.

Achieves top zero-shot performance after human evaluation.

Effective in understanding complex multimodal documents.

Abstract

Vision-language pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text. While existing vision-language pretraining models primarily focus on understanding single image associated with a single piece of text, they often ignore the alignment at the intra-document level, consisting of multiple sentences with multiple images. In this work, we propose DocumentCLIP, a salience-aware contrastive learning framework to enforce vision-language pretraining models to comprehend the interaction between images and longer text within documents. Our model is beneficial for the real-world multimodal document understanding like news article, magazines, product descriptions, which contain linguistically and visually richer content. To the best of our knowledge, we are the first to explore multimodal intra-document links by contrastive learning. In addition, we collect a large Wikipedia dataset for pretraining, which provides various topics and structures. Experiments show DocumentCLIP not only outperforms the state-of-the-art baselines in the supervised setting, but also achieves the best zero-shot performance in the wild after human evaluation. Our code is available at https://github.com/FuxiaoLiu/DocumentCLIP.