Covert Bias: The Severity of Social Views' Unalignment in Language Models Towards Implicit and Explicit Opinion

TL;DR

This paper investigates how large language models handle implicit versus explicit social biases, revealing biases towards explicit opinions and suggesting improvements for model reliability on subjective social topics.

Contribution

It introduces a novel evaluation of bias in LLMs concerning implicit and explicit social opinions and highlights the importance of uncertainty markers for more cautious responses.

Findings

Models show bias towards explicit opinions of opposing stances.

Aligned models generate more cautious responses with uncertainty phrases.

Unaligned models tend to produce incautious, direct responses.

Abstract

While various approaches have recently been studied for bias identification, little is known about how implicit language that does not explicitly convey a viewpoint affects bias amplification in large language models. To examine the severity of bias toward a view, we evaluated the performance of two downstream tasks where the implicit and explicit knowledge of social groups were used. First, we present a stress test evaluation by using a biased model in edge cases of excessive bias scenarios. Then, we evaluate how LLMs calibrate linguistically in response to both implicit and explicit opinions when they are aligned with conflicting viewpoints. Our findings reveal a discrepancy in LLM performance in identifying implicit and explicit opinions, with a general tendency of bias toward explicit opinions of opposing stances. Moreover, the bias-aligned models generate more cautious responses using uncertainty phrases compared to the unaligned (zero-shot) base models. The direct, incautious responses of the unaligned models suggest a need for further refinement of decisiveness by incorporating uncertainty markers to enhance their reliability, especially on socially nuanced topics with high subjectivity.