Deep Semantic Multimodal Hashing Network for Scalable Image-Text and Video-Text Retrievals

TL;DR

This paper introduces DSMHN, a deep multimodal hashing network that efficiently enables scalable image-text and video-text retrieval by jointly learning modality-specific hash functions with semantic preservation.

Contribution

The paper presents a novel deep hashing framework that integrates 2D and 3D CNNs with joint learning of hash functions and semantic labels for improved multimodal retrieval.

Findings

DSMHSN outperforms state-of-the-art methods on multiple datasets.

The framework effectively captures spatial and temporal information.

It demonstrates high retrieval accuracy for both image-text and video-text tasks.

Abstract

Hashing has been widely applied to multimodal retrieval on large-scale multimedia data due to its efficiency in computation and storage. In this article, we propose a novel deep semantic multimodal hashing network (DSMHN) for scalable image-text and video-text retrieval. The proposed deep hashing framework leverages 2-D convolutional neural networks (CNN) as the backbone network to capture the spatial information for image-text retrieval, while the 3-D CNN as the backbone network to capture the spatial and temporal information for video-text retrieval. In the DSMHN, two sets of modality-specific hash functions are jointly learned by explicitly preserving both intermodality similarities and intramodality semantic labels. Specifically, with the assumption that the learned hash codes should be optimal for the classification task, two stream networks are jointly trained to learn the hash functions by embedding the semantic labels on the resultant hash codes. Moreover, a unified deep multimodal hashing framework is proposed to learn compact and high-quality hash codes by exploiting the feature representation learning, intermodality similarity-preserving learning, semantic label-preserving learning, and hash function learning with different types of loss functions simultaneously. The proposed DSMHN method is a generic and scalable deep hashing framework for both image-text and video-text retrievals, which can be flexibly integrated with different types of loss functions. We conduct extensive experiments for both single modal- and cross-modal-retrieval tasks on four widely used multimodal-retrieval data sets. Experimental results on both image-text- and video-text-retrieval tasks demonstrate that the DSMHN significantly outperforms the state-of-the-art methods.

Figures