Language Portability Strategies for Open-domain Dialogue with Pre-trained Language Models from High to Low Resource Languages

TL;DR

This paper investigates strategies to adapt large pre-trained language models for open-domain dialogue in low-resource languages, using French as a case study, and compares different translation and adaptation approaches.

Contribution

It evaluates neural machine translation and adapter-based methods to enhance language portability of dialogue models from high-resource to low-resource languages.

Findings

NMT-based approaches improve low-resource language performance.

Adapter-based fine-tuning with BLOOM enhances dialogue quality.

Strategies enable leveraging multilingual models without extensive new data.

Abstract

In this paper we propose a study of linguistic portability strategies of large pre-trained language models (PLMs) used for open-domain dialogue systems in a high-resource language for this task. In particular the target low-resource language (L_T) will be simulated with French, as it lacks of task-specific resources and allows our human evaluation, when the source language (L_S) is English. For obvious reasons, recent works using such models for open-domain dialogue are mostly developed in English. Yet building specific PLMs for each possible target language supposes collecting new datasets and is costly. For this reason, trying to leverage all existing resources (PLMs and data) in both L_S and L_T , we wish to assess the performance achievable in L_T with different approaches. The first two approaches evaluate the usage of Neural Machine Translation (NMT) at different levels: TrainOnTarget where a L_S dataset is translated before fine-tuning in L_T and TestOnSource where a L_S model is coupled with NMT modules during inference. Then, the advent of BLOOM [2], the world first open-access multilingual large PLM, allow researchers to develop new approaches aiming to leverage not only the model's full accessibility but also its multilingualism and translation abilities. In this context the task is learned in L_S first and adapted to L_T using the MAD-X Adapter architecture [16]. In the two sets of experiments models are evaluated in spoken dialogue conditions with human and the strategies can be compared in terms of perceived interaction quality.