COPAL: Continual Pruning in Large Language Generative Models

TL;DR

COPAL is a novel pruning algorithm that enables efficient continual adaptation of large language models to new domains without retraining, using sensitivity analysis to identify relevant weights.

Contribution

COPAL introduces a sensitivity-guided pruning method for continual domain adaptation in large language models, avoiding costly retraining.

Findings

Outperforms baseline models in efficiency and adaptability

Enables seamless domain adaptation without retraining

Effective across various model sizes

Abstract

Adapting pre-trained large language models to different domains in natural language processing requires two key considerations: high computational demands and model's inability to continual adaptation. To simultaneously address both issues, this paper presents COPAL (COntinual Pruning in Adaptive Language settings), an algorithm developed for pruning large language generative models under a continual model adaptation setting. While avoiding resource-heavy finetuning or retraining, our pruning process is guided by the proposed sensitivity analysis. The sensitivity effectively measures model's ability to withstand perturbations introduced by the new dataset and finds model's weights that are relevant for all encountered datasets. As a result, COPAL allows seamless model adaptation to new domains while enhancing the resource efficiency. Our empirical evaluation on a various size of LLMs show that COPAL outperforms baseline models, demonstrating its efficacy in efficiency and adaptability.