Efficient Test Time Adapter Ensembling for Low-resource Language Varieties

TL;DR

This paper introduces EMEA, a method that ensembles multiple language adapters with optimized weights to improve robustness and performance on low-resource and unseen language varieties in NLP tasks.

Contribution

The paper proposes EMEA, a novel ensemble method that optimizes adapter weights per test sentence to enhance multilingual model robustness without additional training.

Findings

EMEA significantly improves NER and POS tagging across diverse language varieties.

Ensembling adapters increases robustness to unseen languages.

Optimizing ensemble weights per sentence yields better results than fixed weights.

Abstract

Adapters are light-weight modules that allow parameter-efficient fine-tuning of pretrained models. Specialized language and task adapters have recently been proposed to facilitate cross-lingual transfer of multilingual pretrained models (Pfeiffer et al., 2020b). However, this approach requires training a separate language adapter for every language one wishes to support, which can be impractical for languages with limited data. An intuitive solution is to use a related language adapter for the new language variety, but we observe that this solution can lead to sub-optimal performance. In this paper, we aim to improve the robustness of language adapters to uncovered languages without training new adapters. We find that ensembling multiple existing language adapters makes the fine-tuned model significantly more robust to other language varieties not included in these adapters. Building upon this observation, we propose Entropy Minimized Ensemble of Adapters (EMEA), a method that optimizes the ensemble weights of the pretrained language adapters for each test sentence by minimizing the entropy of its predictions. Experiments on three diverse groups of language varieties show that our method leads to significant improvements on both named entity recognition and part-of-speech tagging across all languages.