Exploring Causal Effect of Social Bias on Faithfulness Hallucinations in Large Language Models

TL;DR

This paper investigates how social biases causally influence faithfulness hallucinations in large language models, using causal modeling and a new bias dataset to reveal significant bias effects on hallucination generation.

Contribution

It introduces a causal framework with the Bias Intervention Dataset to measure social bias effects on hallucinations in LLMs, advancing understanding of bias-related hallucinations.

Findings

Bias significantly causes faithfulness hallucinations in LLMs

Different social biases have varying effects on hallucination direction

Bias interventions can reduce unfairness hallucinations

Abstract

Large language models (LLMs) have achieved remarkable success in various tasks, yet they remain vulnerable to faithfulness hallucinations, where the output does not align with the input. In this study, we investigate whether social bias contributes to these hallucinations, a causal relationship that has not been explored. A key challenge is controlling confounders within the context, which complicates the isolation of causality between bias states and hallucinations. To address this, we utilize the Structural Causal Model (SCM) to establish and validate the causality and design bias interventions to control confounders. In addition, we develop the Bias Intervention Dataset (BID), which includes various social biases, enabling precise measurement of causal effects. Experiments on mainstream LLMs reveal that biases are significant causes of faithfulness hallucinations, and the effect of each bias state differs in direction. We further analyze the scope of these causal effects across various models, specifically focusing on unfairness hallucinations, which are primarily targeted by social bias, revealing the subtle yet significant causal effect of bias on hallucination generation.