DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation

TL;DR

DARE is a reinforcement learning framework that adaptively estimates difficulty and co-evolves with the policy to improve training efficiency, response conciseness, and correctness across tasks.

Contribution

It introduces a unified, difficulty-aware RL method that co-evolves difficulty estimation with the policy using importance sampling and adaptive strategies.

Findings

Dare outperforms existing methods in training efficiency and final effectiveness.

It produces more concise responses on easy tasks and improves correctness on hard ones.

The approach is validated across multiple models and domains.

Abstract

Reinforcement learning improves the reasoning ability of large language models but remains costly and sample-inefficient, as many rollouts provide weak learning signals. Difficulty-aware data selection methods attempt to address this by prioritizing moderately difficult prompts, yet our analysis reveals three limitations: difficulty estimates become inaccurate under policy drift, data selection alone yields limited final-performance gains, and inference efficiency remains largely unchanged. These findings suggest that efficient and effective RL requires more than filtering by difficulty: the policy should learn to solve hard tasks while producing concise responses for easy ones. To this end, we propose **Dare**, a unified framework that co-evolves difficulty estimation with the policy via self-normalized importance sampling, maintains diverse difficulty coverage through a symmetric Beta sampling distribution, and applies tailored training strategies across difficulty tiers with adaptive compute allocation. Extensive experiments across multiple models and domains demonstrate that **Dare** consistently outperforms existing methods in training efficiency, final effectiveness, and inference efficiency, producing more concise responses on easy tasks while improving correctness on hard ones. Code is available at https://github.com/EtaYang10th/DARE.