Cross-Modal Coherence for Text-to-Image Retrieval

TL;DR

This paper introduces a coherence-aware model for text-to-image retrieval that explicitly captures the different ways images and text relate, leading to improved retrieval performance and better human preferences.

Contribution

The paper proposes a novel cross-modal coherence model that explicitly models the relations between images and text, enhancing retrieval accuracy and interpretability.

Findings

Models with coherence relations outperform coherence-agnostic models in retrieval tasks.

Human evaluators prefer images retrieved by the coherence-aware model.

Explicit modeling of coherence improves understanding of modality communication.

Abstract

Common image-text joint understanding techniques presume that images and the associated text can universally be characterized by a single implicit model. However, co-occurring images and text can be related in qualitatively different ways, and explicitly modeling it could improve the performance of current joint understanding models. In this paper, we train a Cross-Modal Coherence Modelfor text-to-image retrieval task. Our analysis shows that models trained with image--text coherence relations can retrieve images originally paired with target text more often than coherence-agnostic models. We also show via human evaluation that images retrieved by the proposed coherence-aware model are preferred over a coherence-agnostic baseline by a huge margin. Our findings provide insights into the ways that different modalities communicate and the role of coherence relations in capturing commonsense inferences in text and imagery.