Chip-Tuning: Classify Before Language Models Say

TL;DR

This paper introduces chip-tuning, a structured pruning method for large language models that uses probing classifiers to effectively reduce model size by up to 50% with minimal performance loss.

Contribution

It proposes a novel pruning framework that attaches tiny classifiers to model layers, enabling effective layer removal while maintaining accuracy.

Findings

Chip-tuning achieves up to 50% pruning ratio.

It outperforms previous methods in accuracy and efficiency.

Applicable to multimodal models and compatible with finetuning.

Abstract

The rapid development in the performance of large language models (LLMs) is accompanied by the escalation of model size, leading to the increasing cost of model training and inference. Previous research has discovered that certain layers in LLMs exhibit redundancy, and removing these layers brings only marginal loss in model performance. In this paper, we adopt the probing technique to explain the layer redundancy in LLMs and demonstrate that language models can be effectively pruned with probing classifiers. We propose chip-tuning, a simple and effective structured pruning framework specialized for classification problems. Chip-tuning attaches tiny probing classifiers named chips to different layers of LLMs, and trains chips with the backbone model frozen. After selecting a chip for classification, all layers subsequent to the attached layer could be removed with marginal performance loss. Experimental results on various LLMs and datasets demonstrate that chip-tuning significantly outperforms previous state-of-the-art baselines in both accuracy and pruning ratio, achieving a pruning ratio of up to 50%. We also find that chip-tuning could be applied on multimodal models, and could be combined with model finetuning, proving its excellent compatibility.