Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO

TL;DR

This paper introduces PA-GRPO, a permutation-aware training method for large language models that reduces selection bias by enforcing permutation consistency, leading to improved fairness and performance.

Contribution

The paper proposes a novel permutation-aware training approach, PA-GRPO, that mitigates selection bias in LLMs by enforcing permutation consistency during training.

Findings

PA-GRPO outperforms strong baselines on seven benchmarks.

It substantially reduces selection bias without sacrificing overall performance.

Experimental results validate the effectiveness of permutation-aware optimization.

Abstract

Large language models (LLMs) used for multiple-choice and pairwise evaluation tasks often exhibit selection bias due to non-semantic factors like option positions and label symbols. Existing inference-time debiasing is costly and may harm reasoning, while pointwise training ignores that the same question should yield consistent answers across permutations. To address this issue, we propose Permutation-Aware Group Relative Policy Optimization (PA-GRPO), which mitigates selection bias by enforcing permutation-consistent semantic reasoning. PA-GRPO constructs a permutation group for each instance by generating multiple candidate permutations, and optimizes the model using two complementary mechanisms: (1) cross-permutation advantage, which computes advantages relative to the mean reward over all permutations of the same instance, and (2) consistency-aware reward, which encourages the model to produce consistent decisions across different permutations. Experimental results demonstrate that PA-GRPO outperforms strong baselines across seven benchmarks, substantially reducing selection bias while maintaining high overall performance. The code is available on github (https://github.com/ECNU-Text-Computing/PA-GRPO).