Multi-Modal Mutual Information Maximization: A Novel Approach for Unsupervised Deep Cross-Modal Hashing

TL;DR

This paper introduces a novel unsupervised deep cross-modal hashing method that maximizes mutual information to learn binary representations preserving intra- and inter-modal similarities, improving retrieval performance.

Contribution

The paper proposes CMIMH, a new mutual information maximization approach for unsupervised cross-modal hashing that balances modality gap reduction and private information preservation.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Effectively preserves intra- and inter-modal similarities.

Balances modality gap reduction with private information retention.

Abstract

In this paper, we adopt the maximizing mutual information (MI) approach to tackle the problem of unsupervised learning of binary hash codes for efficient cross-modal retrieval. We proposed a novel method, dubbed Cross-Modal Info-Max Hashing (CMIMH). First, to learn informative representations that can preserve both intra- and inter-modal similarities, we leverage the recent advances in estimating variational lower-bound of MI to maximize the MI between the binary representations and input features and between binary representations of different modalities. By jointly maximizing these MIs under the assumption that the binary representations are modelled by multivariate Bernoulli distributions, we can learn binary representations, which can preserve both intra- and inter-modal similarities, effectively in a mini-batch manner with gradient descent. Furthermore, we find out that trying to minimize the modality gap by learning similar binary representations for the same instance from different modalities could result in less informative representations. Hence, balancing between reducing the modality gap and losing modality-private information is important for the cross-modal retrieval tasks. Quantitative evaluations on standard benchmark datasets demonstrate that the proposed method consistently outperforms other state-of-the-art cross-modal retrieval methods.