Drift to Remember
Jin Du, Xinhe Zhang, Hao Shen, Xun Xian, Ganghua Wang, Jiawei Zhang,, Yuhong Yang, Na Li, Jia Liu, Jie Ding

TL;DR
DriftNet introduces a novel lifelong learning approach inspired by biological neural representational drift, enabling AI models to continually learn new tasks while retaining prior knowledge efficiently, especially in large language models.
Contribution
This paper proposes DriftNet, a scalable neural network that leverages representational drift to improve lifelong learning in AI, particularly in large language models, without full retraining.
Findings
DriftNet outperforms existing models in image classification and NLP tasks.
It effectively handles sequences of tasks like sentiment analysis and question answering.
DriftNet is scalable and efficient for large models on single GPUs.
Abstract
Lifelong learning in artificial intelligence (AI) aims to mimic the biological brain's ability to continuously learn and retain knowledge, yet it faces challenges such as catastrophic forgetting. Recent neuroscience research suggests that neural activity in biological systems undergoes representational drift, where neural responses evolve over time, even with consistent inputs and tasks. We hypothesize that representational drift can alleviate catastrophic forgetting in AI during new task acquisition. To test this, we introduce DriftNet, a network designed to constantly explore various local minima in the loss landscape while dynamically retrieving relevant tasks. This approach ensures efficient integration of new information and preserves existing knowledge. Experimental studies in image classification and natural language processing demonstrate that DriftNet outperforms existing…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications · Topic Modeling
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Attention Is All You Need · Linear Layer · Cosine Annealing · Dense Connections · Multi-Head Attention · Linear Warmup With Linear Decay · Weight Decay · Linear Warmup With Cosine Annealing · Adam
