Identifying and Measuring Token-Level Sentiment Bias in Pre-trained Language Models with Prompts

TL;DR

This paper introduces prompt-based token-level sentiment tests to detect and quantify latent sentiment bias in pre-trained language models, revealing how fine-tuning may increase such biases.

Contribution

It proposes two novel prompt-based tests, SAT and SST, for identifying and measuring sentiment bias in PLMs, advancing bias detection methods.

Findings

SAT and SST effectively identify sentiment bias in PLMs.

SST quantifies the degree of bias.

Fine-tuning can increase existing sentiment biases.

Abstract

Due to the superior performance, large-scale pre-trained language models (PLMs) have been widely adopted in many aspects of human society. However, we still lack effective tools to understand the potential bias embedded in the black-box models. Recent advances in prompt tuning show the possibility to explore the internal mechanism of the PLMs. In this work, we propose two token-level sentiment tests: Sentiment Association Test (SAT) and Sentiment Shift Test (SST) which utilize the prompt as a probe to detect the latent bias in the PLMs. Our experiments on the collection of sentiment datasets show that both SAT and SST can identify sentiment bias in PLMs and SST is able to quantify the bias. The results also suggest that fine-tuning can possibly augment the existing bias in PLMs.