Full-Parameter Continual Pretraining of Gemma2: Insights into Fluency and Domain Knowledge

TL;DR

This paper explores how to improve Lithuanian language fluency in a large language model while preserving existing domain knowledge using continual pretraining and Elastic Weight Consolidation, demonstrating effective adaptation without retraining from scratch.

Contribution

It introduces a method combining continual pretraining with EWC to enhance language fluency and retain domain knowledge in LLMs for under-represented languages.

Findings

EWC mitigates catastrophic forgetting in multilingual LLMs.

The model maintains or improves performance on Lithuanian language tasks.

The approach requires no access to original training data.

Abstract

In this technical report, we empirically investigate the relationship between linguistic fluency and domain knowledge in the context of continual learning with large language models (LLMs). Specifically, we enhance the linguistic fluency of the Gemma2 LLM for the Lithuanian language by autoregressively pretraining its full parameter set on the first 10\% of the Lithuanian language component of the CulturaX dataset. To prevent catastrophic forgetting of the model's existing domain knowledge, we apply Elastic Weight Consolidation (EWC), leveraging Fisher information estimated using data from the Massive Multitask Language Understanding (MMLU) benchmark. In the post-training evaluations, we assess linguistic fluency through perplexity and evaluate domain knowledge using accuracy on a suite of language understanding benchmarks, including ARC-Easy, Belebele, GSM8K, HellaSwag, MMLU, TruthfulQA, and Winogrande, in both English and Lithuanian. The empirical results demonstrate that EWC not only mitigates catastrophic forgetting by preserving the model's performance in terms of both linguistic fluency and domain knowledge but also improves or maintains these capabilities for the newly added Lithuanian language. These findings highlight the potential for more efficient adaptation of general-purpose LLMs to under-represented languages without requiring access to the original training data. The accompanying codebase is openly accessible at https://github.com/Neurotechnology/LLM_EWC.