Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks

TL;DR

This paper introduces SP-AEN, a novel adversarial network that preserves semantics in zero-shot visual recognition, significantly improving accuracy and enabling photo-realistic image generation for unseen classes.

Contribution

The paper proposes a semantics-preserving adversarial embedding network that disentangles and transfers semantic information, addressing semantic loss in zero-shot learning and outperforming prior methods.

Findings

SP-AEN outperforms state-of-the-art methods on four benchmarks.

It improves zero-shot recognition accuracy by up to 12.2%.

It can generate photo-realistic images of unseen classes.

Abstract

We propose a novel framework called Semantics-Preserving Adversarial Embedding Network (SP-AEN) for zero-shot visual recognition (ZSL), where test images and their classes are both unseen during training. SP-AEN aims to tackle the inherent problem --- semantic loss --- in the prevailing family of embedding-based ZSL, where some semantics would be discarded during training if they are non-discriminative for training classes, but could become critical for recognizing test classes. Specifically, SP-AEN prevents the semantic loss by introducing an independent visual-to-semantic space embedder which disentangles the semantic space into two subspaces for the two arguably conflicting objectives: classification and reconstruction. Through adversarial learning of the two subspaces, SP-AEN can transfer the semantics from the reconstructive subspace to the discriminative one, accomplishing the improved zero-shot recognition of unseen classes. Comparing with prior works, SP-AEN can not only improve classification but also generate photo-realistic images, demonstrating the effectiveness of semantic preservation. On four popular benchmarks: CUB, AWA, SUN and aPY, SP-AEN considerably outperforms other state-of-the-art methods by an absolute performance difference of 12.2\%, 9.3\%, 4.0\%, and 3.6\% in terms of harmonic mean values