From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models

TL;DR

This paper investigates how different pruning strategies affect reasoning-augmented large language models, revealing that pruning methods should be tailored to the model's specific reasoning and task characteristics for optimal performance.

Contribution

It provides a controlled, comprehensive study comparing pruning strategies on reasoning-augmented models, highlighting paradigm-dependent effects and guiding better pruning practices.

Findings

Depth pruning outperforms width pruning on classification tasks.

Width pruning is more robust for generation and reasoning tasks.

Static pruning better preserves reasoning performance.

Abstract

Large language models (LLMs) are increasingly costly to deploy, motivating extensive research on model pruning. However, most existing studies focus on instruction-following LLMs, leaving it unclear whether established pruning strategies transfer to reasoning-augmented models that explicitly generate long intermediate reasoning traces. In this work, we conduct a controlled study of pruning for both instruction-following ($\textbf{LLM-instruct}$) and reasoning-augmented ($\textbf{LLM-think}$) models. To isolate the effects of pruning, we align pruning calibration and post-pruning recovery data with each model's original training distribution, which we show yields more stable and reliable pruning behavior. We evaluate static depth pruning, static width pruning, and dynamic pruning across 17 tasks spanning classification, generation, and reasoning. Our results reveal clear paradigm-dependent differences: depth pruning outperforms width pruning on classification tasks, while width pruning is more robust for generation and reasoning. Moreover, static pruning better preserves reasoning performance, whereas dynamic pruning excels on classification and generation but remains challenging for long-chain reasoning. These findings underscore the need for pruning strategies that explicitly account for the distinct characteristics of reasoning-augmented LLMs. Our code is publicly available at https://github.com/EIT-NLP/LRM-Pruning.