Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactuals

TL;DR

This paper introduces a method to improve sentiment classification robustness by automatically generating counterfactual samples from a small manually annotated subset, significantly enhancing out-of-distribution performance.

Contribution

The paper presents a novel approach that leverages automatic counterfactual generation in encoding space, reducing manual annotation needs while improving model robustness.

Findings

Achieved +3% accuracy with only 1% manual counterfactuals

Outperformed methods using 100% in-distribution data

Enhanced OOD test performance on multiple datasets

Abstract

For text classification tasks, finetuned language models perform remarkably well. Yet, they tend to rely on spurious patterns in training data, thus limiting their performance on out-of-distribution (OOD) test data. Among recent models aiming to avoid this spurious pattern problem, adding extra counterfactual samples to the training data has proven to be very effective. Yet, counterfactual data generation is costly since it relies on human annotation. Thus, we propose a novel solution that only requires annotation of a small fraction (e.g., 1%) of the original training data, and uses automatic generation of extra counterfactuals in an encoding vector space. We demonstrate the effectiveness of our approach in sentiment classification, using IMDb data for training and other sets for OOD tests (i.e., Amazon, SemEval and Yelp). We achieve noticeable accuracy improvements by adding only 1% manual counterfactuals: +3% compared to adding +100% in-distribution training samples, +1.3% compared to alternate counterfactual approaches.