Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning

TL;DR

This paper investigates the impact of calibration data choice on large language model pruning, revealing that the commonly used C4 dataset is suboptimal and that alternative datasets can improve pruning efficiency and performance.

Contribution

It systematically evaluates various calibration datasets for LLM pruning, challenging the default use of C4 and highlighting the benefits of arithmetic and downstream datasets.

Findings

C4 is not the optimal calibration dataset for LLM pruning

Arithmetic datasets perform as well or better than pre-training datasets

Downstream datasets do not always improve downstream task performance

Abstract

Network pruning has emerged as a potential solution to make LLMs cheaper to deploy. However, existing LLM pruning approaches universally rely on the C4 dataset as the calibration data for calculating pruning scores, leaving its optimality unexplored. In this study, we evaluate the choice of calibration data on LLM pruning, across a wide range of datasets that are most commonly used in LLM training and evaluation, including four pertaining datasets as well as three categories of downstream tasks encompassing nine datasets. Each downstream dataset is prompted with In-Context Learning (ICL) and Chain-of-Thought (CoT), respectively. Besides the already intriguing observation that the choice of calibration data significantly impacts the performance of pruned LLMs, our results also uncover several subtle and often unexpected findings, summarized as follows: (1) C4 is not the optimal choice for LLM pruning, even among commonly used pre-training datasets; (2) arithmetic datasets, when used as calibration data, performs on par or even better than pre-training datasets; (3) pruning with downstream datasets does not necessarily help the corresponding downstream task, compared to pre-training data; (4) ICL is widely beneficial to all data categories, whereas CoT is only useful on certain tasks. Our findings shed light on the importance of carefully selecting calibration data for LLM pruning and pave the way for more efficient deployment of these powerful models in real-world applications. We release our code at: https://github.com/abx393/llm-pruning-calibration-data.