Overton Pluralistic Reinforcement Learning for Large Language Models

TL;DR

This paper introduces OP-GRPO, a reinforcement learning framework enabling large language models to generate diverse, pluralistic responses reflecting multiple human perspectives without explicit prompts, improving coverage and accuracy.

Contribution

The paper presents a novel RL method that trains models to produce diverse responses aligned with human values, using a similarity estimator and dual-reward system for implicit pluralism.

Findings

Qwen2.5-3B-Instruct surpasses 20B GPT-OSS baseline by 37.4% accuracy.

Model outperforms modular architecture baseline by 19.1%.

Robustness confirmed with evaluations using GPT-4.1.

Abstract

Existing alignment paradigms remain limited in capturing the pluralistic nature of human values. Overton Pluralism addresses this gap by generating responses with diverse perspectives from a single query. This paper introduces OP-GRPO (Overton Pluralistic Group Relative Policy Optimization), a reinforcement learning framework for implicit Overton Pluralism that enables a single large language model to produce pluralistic responses without explicit prompting or modular orchestration. Our workflow consists of two main steps. First, similarity estimator training fine-tunes a Sentence Transformer for Overton Pluralism tasks to provide more accurate coverage evaluation of generated responses. Second, OP-GRPO training incorporates this similarity estimator into a dual-reward system designed to ensure both broad coverage of genuine human perspectives and the uniqueness of each perspective, thereby promoting diversity. Empirical results demonstrate a "small models, big perspective coverage" effect. The trained Qwen2.5-3B-Instruct model surpasses a 20B GPT-OSS baseline with a 37.4 percent relative accuracy gain on a Natural Language Inference benchmark, and also outperforms a modular architecture baseline with a 19.1 percent relative improvement. Additional evaluations using GPT-4.1 as a large language model judge further confirm the robustness of the approach.