Asymmetrically Weighted CCA And Hierarchical Kernel Sentence Embedding For Image & Text Retrieval

TL;DR

This paper introduces an asymmetric weighting scheme for CCA, a computationally efficient model selection method, and a Hierarchical Kernel Sentence Embedding, all enhancing multimodal image and text retrieval performance.

Contribution

It presents a novel asymmetric weighting approach for CCA, an efficient model selection method, and a new hierarchical kernel sentence embedding for improved multimodal retrieval.

Findings

Achieved state-of-the-art results on MSCOCO and Flickr datasets.

Improved retrieval performance through asymmetric weighting of canonical weights.

Enhanced model stability and generalization with spectral filtering-based model selection.

Abstract

Joint modeling of language and vision has been drawing increasing interest. A multimodal data representation allowing for bidirectional retrieval of images by sentences and vice versa is a key aspect. In this paper we present three contributions in canonical correlation analysis (CCA) based multimodal retrieval. Firstly, we show that an asymmetric weighting of the canonical weights, while achieving a cross view mapping from the search to the query space, improves the retrieval performance. Secondly, we devise a computationally efficient model selection, crucial to generalization and stability, in the framework of the Bj\"ork Golub algorithm for regularized CCA via spectral filtering. Finally, we introduce a Hierarchical Kernel Sentence Embedding (HKSE) that approximates Kernel CCA for a special similarity kernel between distribution of words embedded in a vector space. State of the art results are obtained on MSCOCO and Flickr benchmarks when these three techniques are used in conjunction.