Continual-learning for Modelling Low-Resource Languages from Large Language Models

TL;DR

This paper introduces a continual learning approach using POS-based code-switching and replay adapters to reduce catastrophic forgetting when adapting large language models for low-resource languages, demonstrated on vision-language tasks.

Contribution

It proposes a novel continual learning method combining POS-based code-switching and replay adapters to improve low-resource language modeling from large language models.

Findings

Successful mitigation of catastrophic forgetting in low-resource language modeling

Effective application on visual question answering and language modeling tasks

Improved performance over baseline models

Abstract

Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture.