Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation

TL;DR

This paper introduces VMR-RLVR, a novel training strategy that reformulates open-ended tasks into verifiable multiple-choice formats, enabling reinforcement learning with verifiable rewards to improve reasoning in large language models beyond domains with clear solutions.

Contribution

The paper proposes VMR-RLVR, a new method that extends RLVR to open-ended tasks by converting data into verifiable multiple-choice formats, enhancing reasoning capabilities without explicit ground truth.

Findings

Improves LLM performance on open-ended tasks by 3.29 points on average.

Effective across seven diverse open-ended benchmarks.

Enables reinforcement learning without explicit ground truth.

Abstract

Reinforcement Learning with Verifiable Rewards(RLVR) has demonstrated great potential in enhancing the reasoning capabilities of large language models (LLMs). However, its success has thus far been largely confined to the mathematical and programming domains with clear and automatically checkable outcomes. Reinforcement learning on open-ended tasks (e.g., creative writing and subjective Q&A) continues to rely on reward models due to the absence of verifiable solutions. This raises a key question: how can we extend RLVR to strengthen reasoning in open-ended tasks regardless of the absence of the unambiguous ground truth? To overcome this challenge, we introduce Verifiable Multiple-Choice Reformulation for Reinforcement Learning from Verifiable Rewards (VMR-RLVR), a novel training strategy that restructures open-ended data into verifiable multiple-choice formats, enabling effective training even in the absence of explicit ground truth. Experimental results on multiple benchmarks validate the effectiveness of our method in improving LLM performance on open-ended tasks. Notably, across seven open-ended benchmarks, our VMR-RLVR training delivers an average gain of 3.29 points over the RL with reward model.