Dynamic Base model Shift for Delta Compression

TL;DR

This paper introduces Dynamic Base Model Shift (DBMS), a method that adaptively shifts the base model before delta compression to significantly improve performance at high compression ratios across various transformer models.

Contribution

The paper proposes DBMS, a novel approach that dynamically adjusts the base model for delta compression, outperforming existing methods especially at high compression rates.

Findings

DBMS maintains high performance under extreme compression ratios.

It is effective across language, vision, and multi-modal transformer models.

DBMS can be integrated with other compression techniques.

Abstract

Transformer-based models with the pretrain-finetune paradigm bring about significant progress, along with the heavy storage and deployment costs of finetuned models on multiple tasks. Delta compression attempts to lower the costs by reducing the redundancy of delta parameters (i.e., the difference between the finetuned and pre-trained model weights) through pruning or quantization. However, existing methods by default employ the pretrained model as the base model and compress the delta parameters for every task, which may causes significant performance degradation, especially when the compression rate is extremely high. To tackle this issue, we investigate the impact of different base models on the performance of delta compression and find that the pre-trained base model can hardly be optimal. To this end, we propose Dynamic Base Model Shift (DBMS), which dynamically adapts the base model to the target task before performing delta compression. Specifically, we adjust two parameters, which respectively determine the magnitude of the base model shift and the overall scale of delta compression, to boost the compression performance on each task. Through low-cost learning of these two parameters, our DBMS can maintain most of the finetuned model's performance even under an extremely high compression ratio setting, significantly surpassing existing methods. Moreover, our DBMS is orthogonal and can be integrated with a variety of other methods, and it has been evaluated across different types of models including language, vision transformer, and multi-modal models.