Exploring the Efficacy of Automatically Generated Counterfactuals for Sentiment Analysis

TL;DR

This paper presents an automatic method for generating counterfactual data to improve the robustness of sentiment analysis models, reducing reliance on costly human-annotated data.

Contribution

It introduces a novel automated approach for creating counterfactual datasets, enhancing model performance without human-in-the-loop data augmentation.

Findings

Significant performance improvements over models trained on original data.

Outperforms models trained with human-generated augmented data.

Effective across multiple datasets and benchmarks.

Abstract

While state-of-the-art NLP models have been achieving the excellent performance of a wide range of tasks in recent years, important questions are being raised about their robustness and their underlying sensitivity to systematic biases that may exist in their training and test data. Such issues come to be manifest in performance problems when faced with out-of-distribution data in the field. One recent solution has been to use counterfactually augmented datasets in order to reduce any reliance on spurious patterns that may exist in the original data. Producing high-quality augmented data can be costly and time-consuming as it usually needs to involve human feedback and crowdsourcing efforts. In this work, we propose an alternative by describing and evaluating an approach to automatically generating counterfactual data for data augmentation and explanation. A comprehensive evaluation on several different datasets and using a variety of state-of-the-art benchmarks demonstrate how our approach can achieve significant improvements in model performance when compared to models training on the original data and even when compared to models trained with the benefit of human-generated augmented data.