Gradient Projection For Continual Parameter-Efficient Tuning

TL;DR

This paper introduces a unified gradient projection framework called PEGP that enhances parameter-efficient tuning methods by reducing forgetting in continual learning across various models and modalities.

Contribution

It reformulates PETs from a gradient projection perspective and proposes orthogonal gradient projection to effectively mitigate forgetting with minimal additional resources.

Findings

Significantly reduces forgetting in continual learning scenarios.

Effective across diverse models like ViT and CLIP.

Improves generalization in multi-modal and domain adaptation tasks.

Abstract

Parameter-efficient tunings (PETs) have demonstrated impressive performance and promising perspectives in training large models, while they are still confronted with a common problem: the trade-off between learning new content and protecting old knowledge, leading to zero-shot generalization collapse, and cross-modal hallucination. In this paper, we reformulate Adapter, LoRA, Prefix-tuning, and Prompt-tuning from the perspective of gradient projection, and firstly propose a unified framework called Parameter Efficient Gradient Projection (PEGP). We introduce orthogonal gradient projection into different PET paradigms and theoretically demonstrate that the orthogonal condition for the gradient can effectively resist forgetting even for large-scale models. It therefore modifies the gradient towards the direction that has less impact on the old feature space, with less extra memory space and training time. We extensively evaluate our method with different backbones, including ViT and CLIP, on diverse datasets, and experiments comprehensively demonstrate its efficiency in reducing forgetting in class, online class, domain, task, and multi-modality continual settings. The project page is available at https://dmcv-ecnu-pegp.github.io/.