RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling

TL;DR

RuleReasoner introduces a domain-aware dynamic sampling method in reinforcement learning to improve rule-based reasoning across diverse rule formats, achieving superior accuracy and efficiency on multiple benchmarks.

Contribution

It proposes a novel domain-aware dynamic sampling approach in RL for rule reasoning, addressing variability in rule formats and outperforming existing large reasoning models.

Findings

Outperforms frontier LRMs by 4.1% on ID tasks

Achieves 10.4% improvement on OOD tasks

Demonstrates higher computational efficiency

Abstract

Rule-based reasoning is acknowledged as one of the fundamental problems of reasoning. While recent studies show that large reasoning models (LRMs) have remarkable reasoning capabilities enhanced by reinforcement learning (RL), real applications still face severe challenges due to variations in rule formats, types, and complexity. To mitigate this issue, we introduce RuleReasoner, an effective method for rule-based reasoning via a wide collection of curated tasks and a novel domain-aware dynamic sampling approach in RL. Specifically, RuleReasoner resamples each training batch by updating the domain weights based on historical rewards. This facilitates domain balance and active learning schedules for RL, obviating static mix-training engineered by human. Evaluations on in-distribution (ID) and out-of-distribution (OOD) benchmarks reveal that RuleReasoner outperforms frontier LRMs by a significant margin ($\Delta$4.1% on eight ID tasks and $\Delta$10.4% on three OOD tasks over OpenAI-o1). Notably, our approach also exhibits higher computational efficiency compared to prior methods.