Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Models

TL;DR

This paper introduces LOADS, a method that optimizes label set selection for zero-shot classification with large language models by analyzing neuron activation kurtosis, leading to significant performance improvements.

Contribution

The study systematically examines how label design impacts zero-shot ICL and proposes a kurtosis-based method for label selection that enhances performance without additional training.

Findings

Label choice significantly affects model performance and sensitivity.

Optimal labels activate fewer outlier neurons in LLMs.

LOADS improves zero-shot classification accuracy across tasks and languages.

Abstract

In-context learning (ICL) performance is highly sensitive to prompt design, yet the impact of class label options (e.g. lexicon or order) in zero-shot classification remains underexplored. This study proposes LOADS (Label set Optimization via Activation Distribution kurtosiS), a post-hoc method for selecting optimal label sets in zero-shot ICL with large language models (LLMs). LOADS is built upon the observations in our empirical analysis, the first to systematically examine how label option design (i.e., lexical choice, order, and elaboration) impacts classification performance. This analysis shows that the lexical choice of the labels in the prompt (such as agree vs. support in stance classification) plays an important role in both model performance and model's sensitivity to the label order. A further investigation demonstrates that optimal label words tend to activate fewer outlier neurons in LLMs' feed-forward networks. LOADS then leverages kurtosis to measure the neuron activation distribution for label selection, requiring only a single forward pass without gradient propagation or labelled data. The LOADS-selected label words consistently demonstrate effectiveness for zero-shot ICL across classification tasks, datasets, models and languages, achieving maximum performance gain from 0.54 to 0.76 compared to the conventional approach of using original dataset label words.