# Semi-Supervised Cross-Modal Retrieval with Label Prediction

## TL;DR

This paper introduces a deep semi-supervised framework for cross-modal retrieval that predicts labels for unlabeled data and learns a shared representation, outperforming existing methods on standard benchmarks.

## Contribution

It presents a novel deep semi-supervised approach combining label prediction and shared representation learning for cross-modal retrieval.

## Key findings

- Outperforms state-of-the-art in semi-supervised settings
- Effective on multiple benchmark datasets
- Handles both labeled and unlabeled data seamlessly

## Abstract

Due to abundance of data from multiple modalities, cross-modal retrieval tasks with image-text, audio-image, etc. are gaining increasing importance. Of the different approaches proposed, supervised methods usually give significant improvement over their unsupervised counterparts at the additional cost of labeling or annotation of the training data. Semi-supervised methods are recently becoming popular as they provide an elegant framework to balance the conflicting requirement of labeling cost and accuracy. In this work, we propose a novel deep semi-supervised framework which can seamlessly handle both labeled as well as unlabeled data. The network has two important components: (a) the label prediction component predicts the labels for the unlabeled portion of the data and then (b) a common modality-invariant representation is learned for cross-modal retrieval. The two parts of the network are trained sequentially one after the other. Extensive experiments on three standard benchmark datasets, Wiki, Pascal VOC and NUS-WIDE demonstrate that the proposed framework outperforms the state-of-the-art for both supervised and semi-supervised settings.