The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Training

TL;DR

This paper introduces a unified framework for reward structures in training reasoning models, demonstrating that hybrid rewards enhance convergence and stability in fine-tuning large language models on mathematical tasks.

Contribution

It proposes an adaptive hybrid reward scheduler and empirically evaluates various reward formulations, advancing reward design for LLM alignment.

Findings

Hybrid reward structures improve training stability.

Adaptive reward scheduling balances exploration and stability.

Empirical evaluation on GSM8K shows faster convergence.

Abstract

Reward design is central to reinforcement learning from human feedback (RLHF) and alignment research. In this work, we propose a unified framework to study hard, continuous, and hybrid reward structures for fine-tuning large language models (LLMs) on mathematical reasoning tasks. Using Qwen3-4B with LoRA fine-tuning on the GSM8K dataset, we formalize and empirically evaluate reward formulations that incorporate correctness, perplexity, reasoning quality, and consistency. We introduce an adaptive hybrid reward scheduler that transitions between discrete and continuous signals, balancing exploration and stability. Our results show that hybrid reward structures improve convergence speed and training stability over purely hard or continuous approaches, offering insights for alignment via adaptive reward modeling.