Zero-Shot Learning by Convex Combination of Semantic Embeddings

TL;DR

This paper introduces a straightforward zero-shot learning method that combines existing image classifiers with semantic word embeddings using convex combinations, achieving superior results on ImageNet without additional training.

Contribution

It presents a novel, training-free approach to zero-shot learning by leveraging convex combinations of semantic embeddings from existing classifiers and word models.

Findings

Outperforms state-of-the-art zero-shot methods on ImageNet

Requires no additional training beyond existing classifiers and embeddings

Simplifies zero-shot learning with a convex combination approach

Abstract

Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the embedding space is trained jointly with the image transformation. In other cases the semantic embedding space is established by an independent natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional \nway{} classification framing of image understanding, particularly in terms of the promise for zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. In this paper, we propose a simple method for constructing an image embedding system from any existing \nway{} image classifier and a semantic word embedding model, which contains the $\n$ class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional training. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task.