Semantically Grounded Visual Embeddings for Zero-Shot Learning

TL;DR

This paper introduces a joint image-text embedding approach that enhances zero-shot learning by better aligning visual and semantic information through a two-stream network and grounded semantic cues, outperforming existing methods.

Contribution

It proposes a novel joint embedding framework that integrates visual and textual data using a two-stream network and grounded semantic information, improving zero-shot learning performance.

Findings

Improved accuracy on benchmark datasets (+1.6% to +2.6%)

Effective alignment of visual and semantic representations

Enhanced zero-shot recognition performance

Abstract

Zero-shot learning methods rely on fixed visual and semantic embeddings, extracted from independent vision and language models, both pre-trained for other large-scale tasks. This is a weakness of current zero-shot learning frameworks as such disjoint embeddings fail to adequately associate visual and textual information to their shared semantic content. Therefore, we propose to learn semantically grounded and enriched visual information by computing a joint image and text model with a two-stream network on a proxy task. To improve this alignment between image and textual representations, provided by attributes, we leverage ancillary captions to provide grounded semantic information. Our method, dubbed joint embeddings for zero-shot learning is evaluated on several benchmark datasets, improving the performance of existing state-of-the-art methods in both standard ($+1.6$\% on aPY, $+2.6\%$ on FLO) and generalized ($+2.1\%$ on AWA$2$, $+2.2\%$ on CUB) zero-shot recognition.