SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models

TL;DR

The paper introduces SGLP, a novel layer pruning framework guided by representation similarity, which effectively compresses large deep models while preserving their essential features and maintaining high accuracy.

Contribution

It proposes a similarity-guided segmentation and pruning method that considers layer interdependencies, improving compression efficiency without significant performance loss.

Findings

Outperforms state-of-the-art pruning methods in accuracy and efficiency

Achieves significant model compression with minimal performance degradation

Effective on both image classification and large language models

Abstract

Layer pruning has emerged as a potent approach to remove redundant layers in the pre-trained network on the purpose of reducing network size and improve computational efficiency. However, existing layer pruning methods mostly overlook the intrinsic connections and inter-dependencies between different layers within complicated deep neural networks. This oversight can result in pruned models that do not preserve the essential characteristics of the pre-trained network as effectively as desired. To address these limitations, we propose a Similarity-Guided Layer Partition (SGLP) Pruning, a novel pruning framework that exploits representation similarity to guide efficient and informed layer removal for compressing large deep models. Our method begins by employing Centered Kernel Alignment (CKA) to quantify representational similarity between layers, uncovering structural patterns within the network. We then apply Fisher Optimal Segmentation on the similarity matrix to partition the network into semantically coherent layer segments. This segmentation allows pruning decisions to respect layer interdependencies and preserve essential knowledge. Within each segment, we introduce a fine-tuning-free importance evaluation using GradNorm, identifying and removing redundant layers in a targeted, segment-wise manner. Experimental results on both image classification tasks and large language models (LLMs) demonstrate that our proposed SGLP outperforms the state-of-the-art methods in accuracy and efficiency. Our approach achieves significant model compression with minimal performance degradation, making it well-suited for deployment in resource-limited environments.