Latent Embeddings for Zero-shot Classification

TL;DR

This paper introduces a latent embedding model for zero-shot classification that enhances compatibility learning between images and class embeddings, improving accuracy and interpretability over existing methods.

Contribution

It extends bilinear compatibility models by incorporating latent variables to select among multiple maps, leading to better performance and interpretability.

Findings

Improves state-of-the-art accuracy on three datasets.

Provides highly interpretable clustering of object properties.

Uses a ranking-based training objective.

Abstract

We present a novel latent embedding model for learning a compatibility function between image and class embeddings, in the context of zero-shot classification. The proposed method augments the state-of-the-art bilinear compatibility model by incorporating latent variables. Instead of learning a single bilinear map, it learns a collection of maps with the selection, of which map to use, being a latent variable for the current image-class pair. We train the model with a ranking based objective function which penalizes incorrect rankings of the true class for a given image. We empirically demonstrate that our model improves the state-of-the-art for various class embeddings consistently on three challenging publicly available datasets for the zero-shot setting. Moreover, our method leads to visually highly interpretable results with clear clusters of different fine-grained object properties that correspond to different latent variable maps.