Self-calibration for Language Model Quantization and Pruning

TL;DR

This paper introduces self-calibration, a data-free method for quantizing and pruning language models by generating synthetic data from the model itself, improving performance without needing external calibration data.

Contribution

The paper presents a novel self-calibration technique that eliminates the need for external data in model compression, leveraging the model to generate synthetic calibration data.

Findings

Self-calibration often outperforms real data-based calibration methods.

The approach is effective across various models and compression techniques.

Self-calibration maintains high downstream task performance.

Abstract

Quantization and pruning are fundamental approaches for model compression, enabling efficient inference for language models. In a post-training setting, state-of-the-art quantization and pruning methods require calibration data, a small set of unlabeled examples. Conventionally, this is randomly sampled web text, aiming to reflect the model training data. However, this poses two key problems: (1) unrepresentative calibration examples can harm model performance, and (2) organizations increasingly avoid releasing model training data. In this paper, we propose self-calibration as a solution. Our approach requires no external data, instead leveraging the model itself to generate synthetic calibration data, with a view to better approximating the pre-training data distribution. We extensively compare the performance of self-calibration with several baselines, across a variety of models, compression methods, and tasks. Our approach proves consistently competitive in maximizing downstream task performance, frequently outperforming even using real data.