Uncovering the Hidden Cost of Model Compression

TL;DR

This paper investigates how model compression techniques like sparsity and quantization affect the performance and calibration of visual prompting-based transfer learning, revealing that sparsity harms calibration while quantization does not.

Contribution

It provides the first empirical analysis of the impact of model compression on visual prompting transfer, highlighting the differing effects of sparsity and quantization.

Findings

Sparsity negatively impacts model calibration in visual prompting transfer.

Quantization does not adversely affect calibration.

Model compression generally hampers transfer performance, especially with limited data.

Abstract

In an age dominated by resource-intensive foundation models, the ability to efficiently adapt to downstream tasks is crucial. Visual Prompting (VP), drawing inspiration from the prompting techniques employed in Large Language Models (LLMs), has emerged as a pivotal method for transfer learning in the realm of computer vision. As the importance of efficiency continues to rise, research into model compression has become indispensable in alleviating the computational burdens associated with training and deploying over-parameterized neural networks. A primary objective in model compression is to develop sparse and/or quantized models capable of matching or even surpassing the performance of their over-parameterized, full-precision counterparts. Although previous studies have explored the effects of model compression on transfer learning, its impact on visual prompting-based transfer remains unclear. This study aims to bridge this gap, shedding light on the fact that model compression detrimentally impacts the performance of visual prompting-based transfer, particularly evident in scenarios with low data volume. Furthermore, our findings underscore the adverse influence of sparsity on the calibration of downstream visual-prompted models. However, intriguingly, we also illustrate that such negative effects on calibration are not present when models are compressed via quantization. This empirical investigation underscores the need for a nuanced understanding beyond mere accuracy in sparse and quantized settings, thereby paving the way for further exploration in Visual Prompting techniques tailored for sparse and quantized models.