Image-free Classifier Injection for Zero-Shot Classification

TL;DR

This paper introduces ICIS, a method to enable pre-trained classifiers to recognize unseen classes in zero-shot learning without using image data, by injecting classifiers based on class semantics.

Contribution

We propose a novel post-hoc method, ICIS, that injects classifiers into pre-trained models using only class descriptors, eliminating the need for training data for new classes.

Findings

ICIS achieves strong zero-shot classification performance on benchmark datasets.

The method can be trained efficiently and applied directly to existing models.

ICIS outperforms some existing zero-shot learning approaches in generalization.

Abstract

Zero-shot learning models achieve remarkable results on image classification for samples from classes that were not seen during training. However, such models must be trained from scratch with specialised methods: therefore, access to a training dataset is required when the need for zero-shot classification arises. In this paper, we aim to equip pre-trained models with zero-shot classification capabilities without the use of image data. We achieve this with our proposed Image-free Classifier Injection with Semantics (ICIS) that injects classifiers for new, unseen classes into pre-trained classification models in a post-hoc fashion without relying on image data. Instead, the existing classifier weights and simple class-wise descriptors, such as class names or attributes, are used. ICIS has two encoder-decoder networks that learn to reconstruct classifier weights from descriptors (and vice versa), exploiting (cross-)reconstruction and cosine losses to regularise the decoding process. Notably, ICIS can be cheaply trained and applied directly on top of pre-trained classification models. Experiments on benchmark ZSL datasets show that ICIS produces unseen classifier weights that achieve strong (generalised) zero-shot classification performance. Code is available at https://github.com/ExplainableML/ImageFreeZSL .