Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels

TL;DR

This paper introduces NLI-CoAL, a novel bias evaluation method for pre-trained language models in natural language inference that considers all label types, improving bias detection accuracy across multiple languages.

Contribution

It proposes a comprehensive bias measure considering all NLI labels, creates multilingual evaluation datasets, and validates the measure's effectiveness and cross-lingual applicability.

Findings

NLI-CoAL outperforms baseline bias measures in distinguishing biased inferences.

The method is effective across English, Japanese, and Chinese datasets.

First to evaluate PLM bias in Japanese and Chinese NLI tasks.

Abstract

Discriminatory gender biases have been found in Pre-trained Language Models (PLMs) for multiple languages. In Natural Language Inference (NLI), existing bias evaluation methods have focused on the prediction results of one specific label out of three labels, such as neutral. However, such evaluation methods can be inaccurate since unique biased inferences are associated with unique prediction labels. Addressing this limitation, we propose a bias evaluation method for PLMs, called NLI-CoAL, which considers all the three labels of NLI task. First, we create three evaluation data groups that represent different types of biases. Then, we define a bias measure based on the corresponding label output of each data group. In the experiments, we introduce a meta-evaluation technique for NLI bias measures and use it to confirm that our bias measure can distinguish biased, incorrect inferences from non-biased incorrect inferences better than the baseline, resulting in a more accurate bias evaluation. We create the datasets in English, Japanese, and Chinese, and successfully validate the compatibility of our bias measure across multiple languages. Lastly, we observe the bias tendencies in PLMs of different languages. To our knowledge, we are the first to construct evaluation datasets and measure PLMs' bias from NLI in Japanese and Chinese.