Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark

TL;DR

This paper introduces a new benchmark dataset and evaluation framework for natural language-based document image retrieval, enabling fine-grained semantic queries and assessing existing models' zero-shot and fine-tuned performance.

Contribution

It provides a large-scale dataset with high-quality semantic queries and evaluates current models, proposing a two-stage retrieval method for improved efficiency.

Findings

Contrastive vision-language models show promising zero-shot performance.

Fine-tuning enhances retrieval accuracy significantly.

The two-stage retrieval method improves efficiency without sacrificing accuracy.

Abstract

Document image retrieval (DIR) aims to retrieve document images from a gallery according to a given query. Existing DIR methods are primarily based on image queries that retrieve documents within the same coarse semantic category, e.g., newspapers or receipts. However, these methods struggle to effectively retrieve document images in real-world scenarios where textual queries with fine-grained semantics are usually provided. To bridge this gap, we introduce a new Natural Language-based Document Image Retrieval (NL-DIR) benchmark with corresponding evaluation metrics. In this work, natural language descriptions serve as semantically rich queries for the DIR task. The NL-DIR dataset contains 41K authentic document images, each paired with five high-quality, fine-grained semantic queries generated and evaluated through large language models in conjunction with manual verification. We perform zero-shot and fine-tuning evaluations of existing mainstream contrastive vision-language models and OCR-free visual document understanding (VDU) models. A two-stage retrieval method is further investigated for performance improvement while achieving both time and space efficiency. We hope the proposed NL-DIR benchmark can bring new opportunities and facilitate research for the VDU community. Datasets and codes will be publicly available at huggingface.co/datasets/nianbing/NL-DIR.