Collaboratively adding new knowledge to an LLM

TL;DR

This paper explores methods for incrementally adding new knowledge to large language models while preserving existing knowledge, comparing various training strategies in semi-cooperative and fully-cooperative settings.

Contribution

It demonstrates that LoRA-based methods outperform full fine-tuning in knowledge addition and retention, providing practical approaches for continual learning in LLMs.

Findings

LoRA outperforms full fine-tuning in most cases.

Semi-cooperative setting benefits from MOE mixing, model merging, and orthogonal subspace learning.

Joint training and replay are effective in fully-cooperative settings.

Abstract

We address the question of how to successively add new knowledge to an LLM whilst retaining previously-added knowledge. We consider two settings, semi-cooperative and fully-cooperative. Overall, LoRA performs better in most cases than full-fine tuning of all parameters when both new knowledge acquisition and retention of old, including recent, knowledge are taken into account. In the semi-cooperative setting, where datasets are not available after training, MOE mixing, model merging, and LoRA-based orthogonal subspace sequential learning, using a small weight on the orthogonality term, perform well. In the fully-cooperative setting where datasets remain available, joint training and sequential training with replay are both effective approaches with LoRA training generally preferable to full fine-tuning. The codes needed to reproduce the results are provided in an open source repository.