Rethinking Layer Redundancy in Large Language Models: Calibration Objectives and Search for Depth Pruning

TL;DR

This paper investigates how calibration objectives influence depth pruning in large language models, revealing that the choice of objective significantly affects which layers are considered redundant, more than the search algorithm used.

Contribution

It introduces a functional perspective on layer redundancy, emphasizing the impact of calibration objectives over search algorithms in pruning decisions.

Findings

Different calibration objectives lead to distinct pruning patterns.

Perplexity and reasoning accuracy rankings often do not align.

Search algorithms tend to find similar solutions under a fixed objective.

Abstract

Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work has largely treated layer redundancy as an inherent structural property of pretrained networks, emphasizing importance criteria and search algorithms for identifying removable layers. In contrast, we adopt a \emph{functional perspective}, where redundancy depends jointly on the model and the calibration objective, suggesting that a universal layer ranking may not exist. Through an empirical study across three LLM families, two calibration objectives, and seven search algorithms, we find that different objectives produce qualitatively different pruning patterns, while perplexity and downstream reasoning accuracy rankings often fail to align. In contrast, under a fixed objective, different search algorithms tend to converge to similar pruning solutions. Overall, our results suggest that the calibration objective may play a larger role than the particular search algorithm in determining which layers appear redundant.