The Impact of Quantization on Large Reasoning Model Reinforcement Learning

TL;DR

This paper investigates how different quantization methods affect the reasoning performance of large reasoning models trained with reinforcement learning, revealing that quantization-aware training hampers learning while post-training quantization preserves performance.

Contribution

It provides the first systematic analysis of quantization impacts on RL-trained large reasoning models, highlighting the advantages of post-training quantization over quantization-aware training.

Findings

Quantization-aware RL training reduces reasoning performance.

Post-training quantization maintains higher reasoning accuracy.

QLoRA with PTQ outperforms quantization-aware methods.

Abstract

Strong reasoning capabilities can now be achieved by large-scale reinforcement learning (RL) without any supervised fine-tuning. Although post-training quantization (PTQ) and quantization-aware training (QAT) are well studied in the context of fine-tuning, how quantization impacts RL in large reasoning models (LRMs) remains an open question. To answer this question, we conducted systematic experiments and discovered a significant gap in reasoning performance on mathematical benchmarks between post-RL quantized models and their quantization-aware RL optimized counterparts. Our findings suggest that quantization-aware RL training negatively impacted the learning process, whereas PTQ and QLoRA led to greater performance.