Collaborative Parameter Learning: Mitigating Forgetting via Parameter-Level Gradient Analysis

TL;DR

This paper introduces Collaborative Parameter Learning (CPL), a method that selectively updates parameters to reduce catastrophic forgetting in large language models, achieving significant improvements across various tasks.

Contribution

It decomposes gradient similarity into parameter-wise contributions, identifying conflicting and collaborative parameters, and proposes CPL to freeze conflicting ones while updating collaborative ones.

Findings

CPL learns 20.2% to 48.2% more questions with negligible forgetting.

CPL reduces peak VRAM by ~3 GB per billion parameters.

CPL decreases computation time by 16.5%.

Abstract

Catastrophic forgetting during knowledge injection impairs the ability of large language models to acquire new knowledge without overwriting previously mastered knowledge. Recent studies analyze forgetting from a gradient similarity perspective and mitigate forgetting through vector projection. However, these methods primarily characterize gradient similarity at the aggregate direction level, leaving the parameter wise contributions to forgetting underexplored. In this paper, we decompose gradient similarity into parameter wise contributions and identify two types of parameters during forgetting: Conflicting Parameters, whose updates contribute to forgetting and typically account for 50 percent to 75 percent of parameters, and Collaborative Parameters, whose updates mitigate forgetting and account for 25 percent to 50 percent. Based on this analysis, we propose Collaborative Parameter Learning, CPL, a parameter wise training rule that freezes Conflicting Parameters and updates only Collaborative Parameters. Experiments comparing CPL with seven baseline methods show that CPL learns 20.2% to 48.2% more questions with negligible forgetting, while reducing peak VRAM by approximately 3 GB per billion model parameters and computation time by 16.5 percent. Extensive evaluations on parameter consumption, out of set generalization, cross prompt generalization, multimodal tasks, open ended question answering, and multilingual settings demonstrate that CPL effectively mitigates forgetting across diverse scenarios.