FairFlow: An Automated Approach to Model-based Counterfactual Data Augmentation For NLP

TL;DR

FairFlow is an automated, model-based method for generating high-quality counterfactual data to reduce societal biases in NLP models, overcoming limitations of previous dictionary-based approaches.

Contribution

It introduces an automated approach to generate parallel data for counterfactual augmentation, reducing reliance on manual data and improving quality over dictionary-based methods.

Findings

Outperforms dictionary-based substitution in quality and context relevance.

Reduces need for manual parallel data collection.

Effectively mitigates societal biases in NLP models.

Abstract

Despite the evolution of language models, they continue to portray harmful societal biases and stereotypes inadvertently learned from training data. These inherent biases often result in detrimental effects in various applications. Counterfactual Data Augmentation (CDA), which seeks to balance demographic attributes in training data, has been a widely adopted approach to mitigate bias in natural language processing. However, many existing CDA approaches rely on word substitution techniques using manually compiled word-pair dictionaries. These techniques often lead to out-of-context substitutions, resulting in potential quality issues. The advancement of model-based techniques, on the other hand, has been challenged by the need for parallel training data. Works in this area resort to manually generated parallel data that are expensive to collect and are consequently limited in scale. This paper proposes FairFlow, an automated approach to generating parallel data for training counterfactual text generator models that limits the need for human intervention. Furthermore, we show that FairFlow significantly overcomes the limitations of dictionary-based word-substitution approaches whilst maintaining good performance.