MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting

TL;DR

This paper introduces MoE-CT, a novel training architecture for large language models that preserves high-resource language performance while enhancing low-resource language capabilities through a frozen base model and an appended MoE module.

Contribution

The paper presents a new MoE-CT architecture that separates base model training from multilingual expansion, improving low-resource language performance without degrading high-resource language proficiency.

Findings

Outperforms conventional continual training methods in multilingual benchmarks.

Demonstrates enhanced resistance to catastrophic forgetting.

Shows improved transfer learning capabilities.

Abstract

The advent of large language models (LLMs) has predominantly catered to high-resource languages, leaving a disparity in performance for low-resource languages. Conventional Continual Training (CT) approaches to bridge this gap often undermine a model's original linguistic proficiency when expanding to multilingual contexts. Addressing this issue, we introduce a novel MoE-CT architecture, a paradigm that innovatively separates the base model's learning from the multilingual expansion process. Our design freezes the original LLM parameters, thus safeguarding its performance in high-resource languages, while an appended MoE module, trained on diverse language datasets, augments low-resource language proficiency. Our approach significantly outperforms conventional CT methods, as evidenced by our experiments, which show marked improvements in multilingual benchmarks without sacrificing the model's original language performance. Moreover, our MoE-CT framework demonstrates enhanced resistance to forgetting and superior transfer learning capabilities. By preserving the base model's integrity and focusing on strategic parameter expansion, our methodology advances multilingual language modeling and represents a significant step forward for low-resource language inclusion in LLMs, indicating a fruitful direction for future research in language technologies.