Low-Resource Machine Translation through the Lens of Personalized Federated Learning

TL;DR

This paper introduces MeritOpt, a personalized federated learning method for low-resource machine translation that effectively handles heterogeneous data, is interpretable, and demonstrates promising results on Southeast Asian and Finno-Ugric languages.

Contribution

It proposes MeritOpt, a novel personalized federated learning approach tailored for low-resource translation tasks, with enhanced interpretability and minimal interference from unrelated languages.

Findings

Target dataset size influences weight distribution.

Unrelated languages do not interfere with training.

Auxiliary optimizer parameters have minimal impact.

Abstract

We present a new approach called MeritOpt based on the Personalized Federated Learning algorithm MeritFed that can be applied to Natural Language Tasks with heterogeneous data. We evaluate it on the Low-Resource Machine Translation task, using the datasets of South East Asian and Finno-Ugric languages. In addition to its effectiveness, MeritOpt is also highly interpretable, as it can be applied to track the impact of each language used for training. Our analysis reveals that target dataset size affects weight distribution across auxiliary languages, that unrelated languages do not interfere with the training, and auxiliary optimizer parameters have minimal impact. Our approach is easy to apply with a few lines of code, and we provide scripts for reproducing the experiments at https://github.com/VityaVitalich/MeritOpt.