Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillation

TL;DR

This paper addresses the challenge of modal imbalance in vision-language retrieval by proposing a structure-aware distillation method that preserves the geometric structure of representations, leading to improved cross-modal and single-modal retrieval performance.

Contribution

It introduces a novel structure-aware distillation approach for rebalancing cross-modal matching, emphasizing structure preservation to enhance retrieval accuracy.

Findings

Improved cross-modal retrieval performance across multiple datasets.

Enhanced single-modal retrieval capabilities compared to baseline models.

Demonstrated the effectiveness of structure-aware distillation in handling modal imbalance.

Abstract

Vision-language retrieval aims to search for similar instances in one modality based on queries from another modality. The primary objective is to learn cross-modal matching representations in a latent common space. Actually, the assumption underlying cross-modal matching is modal balance, where each modality contains sufficient information to represent the others. However, noise interference and modality insufficiency often lead to modal imbalance, making it a common phenomenon in practice. The impact of imbalance on retrieval performance remains an open question. In this paper, we first demonstrate that ultimate cross-modal matching is generally sub-optimal for cross-modal retrieval when imbalanced modalities exist. The structure of instances in the common space is inherently influenced when facing imbalanced modalities, posing a challenge to cross-modal similarity measurement. To address this issue, we emphasize the importance of meaningful structure-preserved matching. Accordingly, we propose a simple yet effective method to rebalance cross-modal matching by learning structure-preserved matching representations. Specifically, we design a novel multi-granularity cross-modal matching that incorporates structure-aware distillation alongside the cross-modal matching loss. While the cross-modal matching loss constraints instance-level matching, the structure-aware distillation further regularizes the geometric consistency between learned matching representations and intra-modal representations through the developed relational matching. Extensive experiments on different datasets affirm the superior cross-modal retrieval performance of our approach, simultaneously enhancing single-modal retrieval capabilities compared to the baseline models.