Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks

TL;DR

Leopard is a new vision-language model designed for understanding and reasoning across multiple text-rich images, addressing data scarcity and resolution challenges, and outperforming existing models on complex multi-image tasks.

Contribution

We introduce Leopard, a multimodal model with a novel high-resolution multi-image encoding module and a large, high-quality instruction dataset for text-rich, multi-image scenarios.

Findings

Outperforms state-of-the-art models like Llama-3.2 and Qwen2-VL on multi-image benchmarks.

Achieves high performance with only 1.2 million training instances.

Open-sourced code and data facilitate further research.

Abstract

Text-rich images, where text serves as the central visual element guiding the overall understanding, are prevalent in real-world applications, such as presentation slides, scanned documents, and webpage snapshots. Tasks involving multiple text-rich images are especially challenging, as they require not only understanding the content of individual images but reasoning about inter-relationships and logical flows across multiple visual inputs. Despite the importance of these scenarios, current multimodal large language models (MLLMs) struggle to handle such tasks due to two key challenges: (1) the scarcity of high-quality instruction tuning datasets for text-rich multi-image scenarios, and (2) the difficulty in balancing image resolution with visual feature sequence length. To address these challenges, we propose Leopard, an MLLM tailored for handling vision-language tasks involving multiple text-rich images. First, we curated about one million high-quality multimodal instruction-tuning data, tailored to text-rich, multi-image scenarios. Second, we proposed an adaptive high-resolution multi-image encoding module to dynamically optimize the allocation of visual sequence length based on the original aspect ratios and resolutions of images. Experiments on a diverse set of benchmarks reveal that our model consistently outperforms state-of-the-art systems, such as Llama-3.2 and Qwen2-VL, in challenging text-rich, multi-image evaluations. Remarkably, our approach achieves outstanding performance using only 1.2M training instances, all of which are fully open-sourced, demonstrating both high efficiency and effectiveness compared to models trained on large-scale in-house data. Our code and data are available at https://github.com/tencent-ailab/Leopard.