Diversity-Aware Policy Optimization for Large Language Model Reasoning

TL;DR

This paper investigates the role of diversity in reinforcement learning for large language models and introduces a novel diversity-aware policy optimization method that improves reasoning performance and solution diversity.

Contribution

It systematically studies diversity's impact on LLM reasoning and proposes a new method to explicitly promote diversity during RL training.

Findings

Positive correlation between solution diversity and reasoning potential.

3.5% average improvement on mathematical reasoning benchmarks.

Generated solutions are more diverse and robust.

Abstract

The reasoning capabilities of large language models (LLMs) have advanced rapidly, particularly following the release of DeepSeek R1, which has inspired a surge of research into data quality and reinforcement learning (RL) algorithms. Despite the pivotal role diversity plays in RL, its influence on LLM reasoning remains largely underexplored. To bridge this gap, this work presents a systematic investigation into the impact of diversity in RL-based training for LLM reasoning, and proposes a novel diversity-aware policy optimization method. Across evaluations on 12 LLMs, we observe a strong positive correlation between the solution diversity and Potential at k (a novel metric quantifying an LLM's reasoning potential) in high-performing models. This finding motivates our method to explicitly promote diversity during RL training. Specifically, we design a token-level diversity and reformulate it into a practical objective, then we selectively apply it to positive samples. Integrated into the R1-zero training framework, our method achieves a 3.5 percent average improvement across four mathematical reasoning benchmarks, while generating more diverse and robust solutions.