Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?

TL;DR

This paper empirically investigates how different mathematical problem-solving training methods affect large language models' ability to generalize to broader reasoning tasks, highlighting the benefits of long chain-of-thought responses.

Contribution

It systematically compares various MPS training approaches, revealing that long chain-of-thought training and rule-based reinforcement learning significantly improve reasoning generalization.

Findings

Continual pretraining on math text partially generalizes to reasoning tasks.

Instruction tuning on short MPS samples often impairs generalization.

Long chain-of-thought responses enhance reasoning across domains.

Abstract

There has been a growing interest in enhancing the mathematical problem-solving (MPS) capabilities of large language models. While the majority of research efforts concentrate on creating specialized models to solve mathematical problems, it remains unknown how learning mathematical problem-solving generalizes to help develop other reasoning abilities. In this paper, we present an empirical investigation into the generalization potential of various MPS training approaches, such as continual pretraining, instruction tuning, and rule-based reinforcement learning across various data sources, including both short and long chain-of-thought (CoT) samples. Evaluation on 5 mathematical and 8 general reasoning benchmarks show that continual pretraining on math text is able to generalize to general reasoning tasks to some extent. In constrast, instruction tuning on conventional, short MPS samples provides limited benefits and, in many cases, even impairs generalization performance. Notably, training with long CoT responses for MPS samples and incorporating rule-based reinforcement learning on MPS queries exhibit distinct behavior, significantly enhancing generalization by extending the model's reasoning processes into other domains. These results suggest that traditional approaches to learning MPS with short reasoning chains largely fail to achieve robust generalization. However, the emerging paradigm of longer reasoning chains, coupled with self-reflection, offers a promising direction for improving generalized reasoning abilities through learning from specialized domains.