LLM-Driven Completeness and Consistency Evaluation for Cultural Heritage Data Augmentation in Cross-Modal Retrieval

TL;DR

This paper presents $C^3$, a framework that improves cross-modal retrieval of cultural heritage data by enhancing LLM-generated descriptions' completeness and consistency through semantic evaluation and adaptive reasoning supervision.

Contribution

It introduces a novel completeness and consistency evaluation framework for LLM-generated descriptions, improving cross-modal retrieval in cultural heritage datasets.

Findings

$C^3$ achieves state-of-the-art results on cultural heritage datasets.

The framework improves semantic coverage and factual consistency of descriptions.

Enhanced retrieval performance in both fine-tuned and zero-shot settings.

Abstract

Cross-modal retrieval is essential for interpreting cultural heritage data, but its effectiveness is often limited by incomplete or inconsistent textual descriptions, caused by historical data loss and the high cost of expert annotation. While large language models (LLMs) offer a promising solution by enriching textual descriptions, their outputs frequently suffer from hallucinations or miss visually grounded details. To address these challenges, we propose $C^3$, a data augmentation framework that enhances cross-modal retrieval performance by improving the completeness and consistency of LLM-generated descriptions. $C^3$ introduces a completeness evaluation module to assess semantic coverage using both visual cues and language-model outputs. Furthermore, to mitigate factual inconsistencies, we formulate a Markov Decision Process to supervise Chain-of-Thought reasoning, guiding consistency evaluation through adaptive query control. Experiments on the cultural heritage datasets CulTi and TimeTravel, as well as on general benchmarks MSCOCO and Flickr30K, demonstrate that $C^3$ achieves state-of-the-art performance in both fine-tuned and zero-shot settings.