Performance Variability in Zero-Shot Classification

TL;DR

This paper investigates the stability issues of zero-shot classification performance across different class partitions and proposes ensemble learning as a method to reduce variability.

Contribution

It reveals the high variability in ZSC performance under different class splits and introduces ensemble learning to improve stability.

Findings

ZSC performance varies significantly with class partitions

Ensemble learning reduces performance variability

Proposed method enhances robustness of ZSC models

Abstract

Zero-shot classification (ZSC) is the task of learning predictors for classes not seen during training. Although the different methods in the literature are evaluated using the same class splits, little is known about their stability under different class partitions. In this work we show experimentally that ZSC performance exhibits strong variability under changing training setups. We propose the use ensemble learning as an attempt to mitigate this phenomena.