Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop

TL;DR

This paper studies how self-reinforcing feedback loops in large language models influence bias, revealing that such loops can increase preference bias but reduce disparate bias, and proposes a mitigation strategy.

Contribution

It introduces the concept of Self-Consuming Performative Loop (SCPL) and analyzes bias evolution in iterative training with synthetic data, proposing a reward-based rejection sampling method.

Findings

Performative loops increase preference bias.

Performative loops decrease disparate bias.

Rejection sampling mitigates bias effectively.

Abstract

The rapid advancement of large language models (LLMs) has led to growing interest in using synthetic data to train future models. However, this creates a self-consuming retraining loop, where models are trained on their own outputs and may cause performance drops and induce emerging biases. In real-world applications, previously deployed LLMs may influence the data they generate, leading to a dynamic system driven by user feedback. For example, if a model continues to underserve users from a group, less query data will be collected from this particular demographic of users. In this study, we introduce the concept of \textbf{S}elf-\textbf{C}onsuming \textbf{P}erformative \textbf{L}oop (\textbf{SCPL}) and investigate the role of synthetic data in shaping bias during these dynamic iterative training processes under controlled performative feedback. This controlled setting is motivated by the inaccessibility of real-world user preference data from dynamic production systems, and enables us to isolate and analyze feedback-driven bias evolution in a principled manner. We focus on two types of loops, including the typical retraining setting and the incremental fine-tuning setting, which is largely underexplored. Through experiments on three real-world tasks, we find that the performative loop increases preference bias and decreases disparate bias. We design a reward-based rejection sampling strategy to mitigate the bias, moving towards more trustworthy self-improving systems.