Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs

TL;DR

This paper introduces RLDF, a novel bias mitigation method for LLMs that uses multi-role debates among LLMs themselves, replacing human feedback and improving bias reduction.

Contribution

We propose RLDF, a new reinforcement learning approach utilizing multi-role debates among LLMs for bias mitigation, eliminating the need for human feedback.

Findings

RLDF effectively reduces bias in LLMs.

Multi-role debates enhance bias recognition and mitigation.

The approach outperforms traditional bias mitigation methods.

Abstract

Bias in LLMs can harm user experience and societal outcomes. However, current bias mitigation methods often require intensive human feedback, lack transferability to other topics or yield overconfident and random outputs. We find that involving LLMs in role-playing scenario boosts their ability to recognize and mitigate biases. Based on this, we propose Reinforcement Learning from Multi-role Debates as Feedback (RLDF), a novel approach for bias mitigation replacing human feedback in traditional RLHF. We utilize LLMs in multi-role debates to create a dataset that includes both high-bias and low-bias instances for training the reward model in reinforcement learning. Our approach comprises two modes: (1) self-reflection, where the same LLM participates in multi-role debates, and (2) teacher-student, where a more advanced LLM like GPT-3.5-turbo guides the LLM to perform this task. Experimental results across different LLMs on BBQ and our datasets demonstrate the effectiveness of our approach in bias mitigation. Our source code and datasets are available at \texttt{https://anonymous.4open.science/r/RLDF-E344}.