Unsupervised Ranking and Aggregation of Label Descriptions for Zero-Shot Classifiers

TL;DR

This paper introduces an unsupervised method for ranking and aggregating label descriptions to improve zero-shot text classification, leveraging probabilistic models to select optimal descriptions and combine multiple noisy labels for better accuracy.

Contribution

It proposes a novel unsupervised approach using probabilistic models for selecting and aggregating label descriptions in zero-shot classifiers, addressing the challenge of label design without a development set.

Findings

The method effectively ranks label descriptions across diverse datasets.

Aggregating multiple noisy descriptions enhances classification performance.

The approach outperforms baseline methods in zero-shot settings.

Abstract

Zero-shot text classifiers based on label descriptions embed an input text and a set of labels into the same space: measures such as cosine similarity can then be used to select the most similar label description to the input text as the predicted label. In a true zero-shot setup, designing good label descriptions is challenging because no development set is available. Inspired by the literature on Learning with Disagreements, we look at how probabilistic models of repeated rating analysis can be used for selecting the best label descriptions in an unsupervised fashion. We evaluate our method on a set of diverse datasets and tasks (sentiment, topic and stance). Furthermore, we show that multiple, noisy label descriptions can be aggregated to boost the performance.