Long-tail Cross Modal Hashing

TL;DR

This paper introduces LtCMH, a novel cross-modal hashing method designed to effectively handle imbalanced long-tail multi-modal data by mining individuality and commonality features to improve retrieval performance.

Contribution

LtCMH uniquely combines auto-encoders with dynamic feature integration to address long-tail imbalanced multi-modal data in cross-modal hashing.

Findings

Outperforms state-of-the-art methods on long-tail datasets

Maintains comparable performance on balanced datasets

Effectively mines individuality and commonality features

Abstract

Existing Cross Modal Hashing (CMH) methods are mainly designed for balanced data, while imbalanced data with long-tail distribution is more general in real-world. Several long-tail hashing methods have been proposed but they can not adapt for multi-modal data, due to the complex interplay between labels and individuality and commonality information of multi-modal data. Furthermore, CMH methods mostly mine the commonality of multi-modal data to learn hash codes, which may override tail labels encoded by the individuality of respective modalities. In this paper, we propose LtCMH (Long-tail CMH) to handle imbalanced multi-modal data. LtCMH firstly adopts auto-encoders to mine the individuality and commonality of different modalities by minimizing the dependency between the individuality of respective modalities and by enhancing the commonality of these modalities. Then it dynamically combines the individuality and commonality with direct features extracted from respective modalities to create meta features that enrich the representation of tail labels, and binaries meta features to generate hash codes. LtCMH significantly outperforms state-of-the-art baselines on long-tail datasets and holds a better (or comparable) performance on datasets with balanced labels.