OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

TL;DR

OpenDelta is a flexible, plug-and-play library that simplifies parameter-efficient adaptation of large pre-trained models without modifying their code, enabling easier experimentation and deployment.

Contribution

We introduce OpenDelta, a modular library that allows parameter-efficient tuning of PTMs without code modifications, enhancing flexibility and usability.

Findings

Supports multiple delta tuning methods

Compatible with various PTMs without code changes

Simplifies and accelerates model adaptation workflows

Abstract

The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as "delta tuning", which updates only a small subset of parameters, known as "delta modules", while keeping the backbone model's parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.