Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solution

TL;DR

This paper introduces CulTi, a specialized dataset for Chinese cultural heritage, and proposes LACLIP, a local alignment method that improves cross-modal retrieval of intricate visual and textual Chinese heritage data.

Contribution

The paper provides the first dedicated Chinese cultural heritage dataset CulTi and develops LACLIP, a novel training-free local alignment strategy for enhanced cross-modal retrieval.

Findings

LACLIP outperforms existing models in cross-modal retrieval accuracy.

CulTi dataset presents unique challenges due to intricate visual-textual alignment.

LACLIP effectively handles fine-grained semantic associations in Chinese heritage data.

Abstract

China has a long and rich history, encompassing a vast cultural heritage that includes diverse multimodal information, such as silk patterns, Dunhuang murals, and their associated historical narratives. Cross-modal retrieval plays a pivotal role in understanding and interpreting Chinese cultural heritage by bridging visual and textual modalities to enable accurate text-to-image and image-to-text retrieval. However, despite the growing interest in multimodal research, there is a lack of specialized datasets dedicated to Chinese cultural heritage, limiting the development and evaluation of cross-modal learning models in this domain. To address this gap, we propose a multimodal dataset named CulTi, which contains 5,726 image-text pairs extracted from two series of professional documents, respectively related to ancient Chinese silk and Dunhuang murals. Compared to existing general-domain multimodal datasets, CulTi presents a challenge for cross-modal retrieval: the difficulty of local alignment between intricate decorative motifs and specialized textual descriptions. To address this challenge, we propose LACLIP, a training-free local alignment strategy built upon a fine-tuned Chinese-CLIP. LACLIP enhances the alignment of global textual descriptions with local visual regions by computing weighted similarity scores during inference. Experimental results on CulTi demonstrate that LACLIP significantly outperforms existing models in cross-modal retrieval, particularly in handling fine-grained semantic associations within Chinese cultural heritage.