Representation Finetuning for Continual Learning
Haihua Luo, Xuming Ran, Tommi K\"arkk\"ainen, Huiyan Xue, Zhonghua Chen, Qi Xu, Fengyu Cong
TL;DR
This paper introduces CoRe, a novel representation finetuning framework that enhances continual learning by explicitly controlling representation drift, leading to improved stability and plasticity with parameter efficiency.
Contribution
CoRe shifts finetuning from weight space to representation space, using low-rank subspace interventions with explicit objectives for better continual learning performance.
Findings
Outperforms state-of-the-art methods on multiple benchmarks.
Maintains stability for past tasks while adapting to new ones.
Achieves high parameter efficiency through low-rank updates.
Abstract
The world is inherently dynamic, and continual learning aims to enable models to adapt to ever-evolving data streams. While pre-trained models have shown powerful performance in continual learning, they still require finetuning to adapt effectively to downstream tasks. However, prevailing Parameter-Efficient Fine-Tuning (PEFT) methods operate through empirical, black-box optimization at the weight level. These approaches lack explicit control over representation drift, leading to sensitivity to domain shifts and catastrophic forgetting in continual learning scenarios. In this work, we introduce Continual Representation Learning (CoRe), a novel framework that for the first time shifts the finetuning paradigm from weight space to representation space. Unlike conventional methods, CoRe performs task-specific interventions within a low-rank linear subspace of hidden representations,…
Peer Reviews
Decision·ICLR 2026 Conference Withdrawn Submission
+ The paper is well-written, clearly structured, and easy to follow. Figures and tables are informative and support the narrative effectively. The method is explained with sufficient detail, and the experimental setup is thoroughly described to ensure reproducibility.
+ No state-of-the-art CL methods are compared, such as L2P, Dulaprompt, Codaprompt, Inflora, sdlora, hideprompt, etc. + CIFAR is already included in the ViT pre-trained data, it is better to use dataset like imagenet-r for experiments sec. 4.3-sec. 4.4.
(1) The manuscript introduces representation finetuning methods into continual learning framework. The content is clearly presented and well structured, enhancing the clarity and readability of this manuscript. (2) Experiments are conducted across several public datasets, analyzing the effects of subspace rank and data imbalance under different scenarios to evaluate the effectiveness of the proposed method.
(1) The analysis of existing pre-trained continual learning methods is insufficient. The authors introduce three typical continual learning methods in the related work. It is suggested to include a comparative analysis of the advantages and limitations of existing methods based on pre-trained models, providing a clearer perspective on pre-training-based continual learning methods. (2) The experimental comparisons do not include the latest methods. In Section 4, the authors compare three paramet
1. This manuscript considered the perspective of the counterfactual intervention, which is interesting and less explored in the context of CL-PTM.
1. The motivation of the proposed method is not clear. I didn't get why the descriptions in Sections 3.2 & 3.3 were introduced. 2. Some basic concepts of continual learning in Section 3.1 were not accurate, or even wrong. 3. The presentation of this paper was poor. The context of Domain Intervention Interpretation (DII, shown in Section 3.2), which should be the theoretical foundation of this paper, was not well interpreted. The concept was not clear, and the relevant existing studies was not di
The paper addresses the finetuning techniques in continual learning with an emphasis on low rank adaptation. The paper adopted the representation finetuning methods into continual learning scenario and provided comparisons with existing methods to validate its effectiveness.
1. The method is applying an existing representation finetuning method to the continual learning scenario, the contribution is too incremental. 2. There is no clear motivation for applying the specific finetuning method.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Advanced Neural Network Applications · Human Pose and Action Recognition