Compressed models are NOT miniature versions of large models

TL;DR

This paper challenges the assumption that compressed neural models are miniature versions of large models, showing they differ significantly in various characteristics and have major side effects.

Contribution

The study systematically compares compressed models with large models across multiple characteristics, revealing significant differences and questioning their equivalence.

Findings

Compressed models differ significantly from large models in prediction errors.

Compressed models show different data representations and vulnerabilities.

Compressed models exhibit major side effects beyond performance loss.

Abstract

Large neural models are often compressed before deployment. Model compression is necessary for many practical reasons, such as inference latency, memory footprint, and energy consumption. Compressed models are assumed to be miniature versions of corresponding large neural models. However, we question this belief in our work. We compare compressed models with corresponding large neural models using four model characteristics: prediction errors, data representation, data distribution, and vulnerability to adversarial attack. We perform experiments using the BERT-large model and its five compressed versions. For all four model characteristics, compressed models significantly differ from the BERT-large model. Even among compressed models, they differ from each other on all four model characteristics. Apart from the expected loss in model performance, there are major side effects of using compressed models to replace large neural models.