Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration

TL;DR

This paper introduces a novel calibration network called FACTUAL that uses counterfactual augmented data to mitigate biases in large language models for stance detection, improving accuracy and generalization.

Contribution

The paper proposes a new debiasing method, Counterfactual Augmented Calibration Network (FACTUAL), which enhances stance detection by reducing biases in LLMs through counterfactual data.

Findings

FACTUAL effectively reduces stance bias in LLMs.

The method achieves state-of-the-art results on stance detection tasks.

Improves out-of-domain generalization of stance detection models.

Abstract

Stance detection is critical for understanding the underlying position or attitude expressed toward a topic. Large language models (LLMs) have demonstrated significant advancements across various natural language processing tasks including stance detection, however, their performance in stance detection is limited by biases and spurious correlations inherent due to their data-driven nature. Our statistical experiment reveals that LLMs are prone to generate biased stances due to sentiment-stance spurious correlations and preference towards certain individuals and topics. Furthermore, the results demonstrate a strong negative correlation between stance bias and stance detection performance, underscoring the importance of mitigating bias to enhance the utility of LLMs in stance detection. Therefore, in this paper, we propose a Counterfactual Augmented Calibration Network (FACTUAL), which a novel calibration network is devised to calibrate potential bias in the stance prediction of LLMs. Further, to address the challenge of effectively learning bias representations and the difficulty in the generalizability of debiasing, we construct counterfactual augmented data. This approach enhances the calibration network, facilitating the debiasing and out-of-domain generalization. Experimental results on in-target and zero-shot stance detection tasks show that the proposed FACTUAL can effectively mitigate biases of LLMs, achieving state-of-the-art results.