Adapting Language Models to Indonesian Local Languages: An Empirical Study of Language Transferability on Zero-Shot Settings

TL;DR

This study evaluates how well pre-trained language models transfer to low-resource Indonesian local languages in zero-shot sentiment analysis, highlighting the importance of prior language exposure and the effectiveness of adapter-based methods.

Contribution

It provides an empirical comparison of monolingual, multilingual, and adapter-based models on Indonesian local languages, revealing transferability patterns and improving performance with MAD-X.

Findings

Multilingual models excel on seen languages

MAD-X enhances zero-shot transfer performance

Prior language exposure predicts transfer success

Abstract

In this paper, we investigate the transferability of pre-trained language models to low-resource Indonesian local languages through the task of sentiment analysis. We evaluate both zero-shot performance and adapter-based transfer on ten local languages using models of different types: a monolingual Indonesian BERT, multilingual models such as mBERT and XLM-R, and a modular adapter-based approach called MAD-X. To better understand model behavior, we group the target languages into three categories: seen (included during pre-training), partially seen (not included but linguistically related to seen languages), and unseen (absent and unrelated in pre-training data). Our results reveal clear performance disparities across these groups: multilingual models perform best on seen languages, moderately on partially seen ones, and poorly on unseen languages. We find that MAD-X significantly improves performance, especially for seen and partially seen languages, without requiring labeled data in the target language. Additionally, we conduct a further analysis on tokenization and show that while subword fragmentation and vocabulary overlap with Indonesian correlate weakly with prediction quality, they do not fully explain the observed performance. Instead, the most consistent predictor of transfer success is the model's prior exposure to the language, either directly or through a related language.