Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMs

TL;DR

This paper introduces Sliding-Window Merging, a dynamic compression technique that merges similar consecutive layers in large language models to reduce redundancy and maintain performance, outperforming existing pruning methods.

Contribution

The paper presents a novel layer merging approach based on functional similarity, effectively simplifying LLMs while preserving their inference capabilities.

Findings

Outperforms existing pruning techniques in zero-shot inference.

Achieves 1.654% performance improvement with 35% pruning on Vicuna-7B.

Demonstrates potential of combining depth and width pruning.

Abstract

Depth-wise pruning accelerates LLM inference in resource-constrained scenarios but suffers from performance degradation due to direct removal of entire Transformer layers. This paper reveals ``Patch-like'' redundancy across layers via correlation analysis of the outputs of different layers in reproducing kernel Hilbert space, demonstrating consecutive layers exhibit high functional similarity. Building on this observation, this paper proposes Sliding-Window Merging (SWM) - a dynamic compression method that selects consecutive layers from top to bottom using a pre-defined similarity threshold, and compacts patch-redundant layers through a parameter consolidation, thereby simplifying the model structure while maintaining its performance. Extensive experiments on LLMs with various architectures and different parameter scales show that our method outperforms existing pruning techniques in both zero-shot inference performance and retraining recovery quality after pruning. In particular, in the experiment with 35% pruning on the Vicuna-7B model, our method achieved a 1.654% improvement in average performance on zero-shot tasks compared to the existing method. Moreover, we further reveal the potential of combining depth pruning with width pruning to enhance the pruning effect. Our codes are available at https://github.com/920927/SLM-a-sliding-layer-merging-method.