Improving Commonsense Bias Classification by Mitigating the Influence of Demographic Terms

TL;DR

This paper proposes three methods to reduce demographic bias in commonsense classification tasks, significantly improving model accuracy by mitigating demographic term influence through hierarchical generalization and augmentation techniques.

Contribution

It introduces a novel combined approach (IHTA) that effectively mitigates demographic bias, enhancing classifier performance beyond existing augmentation methods.

Findings

Hierarchical generalization increases accuracy by 2.33%.

Threshold-based augmentation improves accuracy by 0.96%.

IHTA achieves 8.82% and 9.96% higher accuracy than baseline methods.

Abstract

Understanding commonsense knowledge is crucial in the field of Natural Language Processing (NLP). However, the presence of demographic terms in commonsense knowledge poses a potential risk of compromising the performance of NLP models. This study aims to investigate and propose methods for enhancing the performance and effectiveness of a commonsense polarization classifier by mitigating the influence of demographic terms. Three methods are introduced in this paper: (1) hierarchical generalization of demographic terms (2) threshold-based augmentation and (3) integration of hierarchical generalization and threshold-based augmentation methods (IHTA). The first method involves replacing demographic terms with more general ones based on a term hierarchy ontology, aiming to mitigate the influence of specific terms. To address the limited bias-related information, the second method measures the polarization of demographic terms by comparing the changes in the model's predictions when these terms are masked versus unmasked. This method augments commonsense sentences containing terms with high polarization values by replacing their predicates with synonyms generated by ChatGPT. The third method combines the two approaches, starting with threshold-based augmentation followed by hierarchical generalization. The experiments show that the first method increases the accuracy over the baseline by 2.33%, and the second one by 0.96% over standard augmentation methods. The IHTA techniques yielded an 8.82% and 9.96% higher accuracy than threshold-based and standard augmentation methods, respectively.