CCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class Classification

TL;DR

CCPrefix introduces a novel prefix-tuning method for many-class classification that uses counterfactuals to address verbalizer ambiguity, improving performance in both fully supervised and few-shot settings.

Contribution

The paper proposes CCPrefix, a new prefix-tuning approach utilizing counterfactuals to enhance many-class classification, overcoming verbalizer ambiguity issues.

Findings

Outperforms previous baselines on benchmark datasets.

Effective in both fully supervised and few-shot scenarios.

Addresses verbalizer ambiguity in many-class classification.

Abstract

Recently, prefix-tuning was proposed to efficiently adapt pre-trained language models to a broad spectrum of natural language classification tasks. It leverages soft prefix as task-specific indicators and language verbalizers as categorical-label mentions to narrow the formulation gap from pre-training language models. However, when the label space increases considerably (i.e., many-class classification), such a tuning technique suffers from a verbalizer ambiguity problem since the many-class labels are represented by semantic-similar verbalizers in short language phrases. To overcome this, inspired by the human-decision process that the most ambiguous classes would be mulled over for each instance, we propose a brand-new prefix-tuning method, Counterfactual Contrastive Prefix-tuning (CCPrefix), for many-class classification. Basically, an instance-dependent soft prefix, derived from fact-counterfactual pairs in the label space, is leveraged to complement the language verbalizers in many-class classification. We conduct experiments on many-class benchmark datasets in both the fully supervised setting and the few-shot setting, which indicates that our model outperforms former baselines.