Graph Pattern Loss based Diversified Attention Network for Cross-Modal Retrieval

TL;DR

This paper introduces GPLDAN, an unsupervised cross-modal retrieval model that uses diversified attention and a novel graph pattern loss to analyze and leverage correlations among multi-modal representations, improving retrieval performance.

Contribution

The paper proposes a new Graph Pattern Loss and diversified attention mechanism for unsupervised cross-modal retrieval, enhancing correlation analysis among representations.

Findings

GPLDAN outperforms state-of-the-art methods on four datasets.

The graph pattern loss effectively captures correlations among different representations.

Diversified attention improves the discrimination of multi-modal features.

Abstract

Cross-modal retrieval aims to enable flexible retrieval experience by combining multimedia data such as image, video, text, and audio. One core of unsupervised approaches is to dig the correlations among different object representations to complete satisfied retrieval performance without requiring expensive labels. In this paper, we propose a Graph Pattern Loss based Diversified Attention Network(GPLDAN) for unsupervised cross-modal retrieval to deeply analyze correlations among representations. First, we propose a diversified attention feature projector by considering the interaction between different representations to generate multiple representations of an instance. Then, we design a novel graph pattern loss to explore the correlations among different representations, in this graph all possible distances between different representations are considered. In addition, a modality classifier is added to explicitly declare the corresponding modalities of features before fusion and guide the network to enhance discrimination ability. We test GPLDAN on four public datasets. Compared with the state-of-the-art cross-modal retrieval methods, the experimental results demonstrate the performance and competitiveness of GPLDAN.