Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following

TL;DR

This paper introduces a self-supervised reinforcement learning framework that enhances reasoning models' instruction following abilities using internal signals, avoiding external supervision and reducing costs.

Contribution

It presents a novel self-supervised RL method that improves instruction following in reasoning models without relying on external models or supervision.

Findings

Significant improvement in instruction following capabilities.

Maintains reasoning performance after enhancement.

Offers a scalable, cost-effective solution.

Abstract

Reasoning models excel in complex problem solving but exhibit a concerning trade off between reasoning capabilities and instruction following abilities. Existing approaches for improving instruction following rely on stronger external models, creating methodological bottlenecks and practical limitations including increased costs and accessibility constraints. We propose a self-supervised RL framework that leverages reasoning models' own internal signals to improve instruction following capabilities without external supervision. Extensive experiments demonstrate that our framework significantly improves instruction following capabilities while maintaining reasoning performance, offering a scalable and cost-effective approach to enhance instruction following in reasoning models. The data and code are publicly available at https://github.com/Rainier-rq/verl-if.