Integrating Information Theory and Adversarial Learning for Cross-modal Retrieval

TL;DR

This paper introduces a novel cross-modal retrieval method that combines information theory and adversarial learning to bridge heterogeneity and semantic gaps between visual and textual data, improving feature alignment.

Contribution

It proposes an integrated framework using Shannon information entropy and adversarial training to reduce distribution and semantic gaps in cross-modal retrieval.

Findings

Effective in reducing modality distribution discrepancy.

Improves intra- and inter-modality feature similarity.

Validated on four benchmarks with four deep models.

Abstract

Accurately matching visual and textual data in cross-modal retrieval has been widely studied in the multimedia community. To address these challenges posited by the heterogeneity gap and the semantic gap, we propose integrating Shannon information theory and adversarial learning. In terms of the heterogeneity gap, we integrate modality classification and information entropy maximization adversarially. For this purpose, a modality classifier (as a discriminator) is built to distinguish the text and image modalities according to their different statistical properties. This discriminator uses its output probabilities to compute Shannon information entropy, which measures the uncertainty of the modality classification it performs. Moreover, feature encoders (as a generator) project uni-modal features into a commonly shared space and attempt to fool the discriminator by maximizing its output information entropy. Thus, maximizing information entropy gradually reduces the distribution discrepancy of cross-modal features, thereby achieving a domain confusion state where the discriminator cannot classify two modalities confidently. To reduce the semantic gap, Kullback-Leibler (KL) divergence and bi-directional triplet loss are used to associate the intra- and inter-modality similarity between features in the shared space. Furthermore, a regularization term based on KL-divergence with temperature scaling is used to calibrate the biased label classifier caused by the data imbalance issue. Extensive experiments with four deep models on four benchmarks are conducted to demonstrate the effectiveness of the proposed approach.