DARL: Encouraging Diverse Answers for General Reasoning without Verifiers

TL;DR

DARL is a reinforcement learning framework that promotes diverse, high-quality answers in large language models without relying on domain-specific verifiers, improving reasoning and output variety.

Contribution

DARL introduces a simple, effective method to encourage answer diversity in general reasoning tasks without extra verifiers, compatible with existing RL approaches.

Findings

DARL outperforms RLPR on multiple benchmarks.

Achieves 1.3-point average gain on reasoning benchmarks.

Achieves 9.5-point average gain on general benchmarks.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated promising gains in enhancing the reasoning capabilities of large language models. However, its dependence on domain-specific verifiers significantly restricts its applicability to open and general domains. Recent efforts such as RLPR have extended RLVR to general domains, enabling training on broader datasets and achieving improvements over RLVR. However, a notable limitation of these methods is their tendency to overfit to reference answers, which constrains the model's ability to generate diverse outputs. This limitation is particularly pronounced in open-ended tasks such as writing, where multiple plausible answers exist. To address this, we propose DARL, a simple yet effective reinforcement learning framework that encourages the generation of diverse answers within a controlled deviation range from the reference while preserving alignment with it. Our framework is fully compatible with existing general reinforcement learning methods and can be seamlessly integrated without additional verifiers. Extensive experiments on thirteen benchmarks demonstrate consistent improvements in reasoning performance. Notably, DARL surpasses RLPR, achieving average gains of 1.3 points on six reasoning benchmarks and 9.5 points on seven general benchmarks, highlighting its effectiveness in improving both reasoning accuracy and output diversity.