Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models

TL;DR

This paper introduces a novel calibration strategy using self-generated reasoning data to improve pruning of large reasoning models, significantly enhancing their reasoning capabilities after pruning.

Contribution

It is the first empirical study on pruning large reasoning models and proposes a selective data construction method for better calibration during pruning.

Findings

Calibration with self-generated reasoning data improves pruning performance by 10-13%.

Challenging and moderately long reasoning data are most effective for calibration.

Existing pruning methods are less effective without specialized calibration data.

Abstract

Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex reasoning benchmarks. However, their long chain-of-thought reasoning processes incur significant inference overhead. Pruning has emerged as a promising approach to reducing computational costs. However, existing efforts have primarily focused on large language models (LLMs), while pruning LRMs remains unexplored. In this work, we conduct the first empirical study on pruning LRMs and show that directly applying existing pruning techniques fails to yield satisfactory results. Our findings indicate that using self-generated reasoning data for calibration can substantially improve pruning performance. We further investigate how the difficulty and length of reasoning data affect pruning outcomes. Our analysis reveals that challenging and moderately long self-generated reasoning data serve as ideal calibration data. Based on these insights, we propose a Selective Self-Generated Reasoning (SSGR) data construction strategy to provide effective calibration data for pruning LRMs. Experimental results on the DeepSeek-R1-Distill model series validate that our strategy improves the reasoning ability of pruned LRMs by 10%-13% compared to general pruning methods.