Limited-Resource Adapters Are Regularizers, Not Linguists

TL;DR

This paper investigates adapter methods for low-resource machine translation of Creole languages, revealing that adapters mainly act as regularizers rather than transferring linguistic knowledge, with performance not tied to linguistic relatedness.

Contribution

It demonstrates that adapter-based fine-tuning's effectiveness in low-resource MT is primarily due to regularization, challenging assumptions about linguistic transfer.

Findings

Adapters improve low-resource MT performance but are not linked to linguistic relatedness.

Randomly initialized adapters perform similarly to trained ones, indicating a regularization effect.

Linguistic relatedness does not predict adapter effectiveness in this setting.

Abstract

Cross-lingual transfer from related high-resource languages is a well-established strategy to enhance low-resource language technologies. Prior work has shown that adapters show promise for, e.g., improving low-resource machine translation (MT). In this work, we investigate an adapter souping method combined with cross-attention fine-tuning of a pre-trained MT model to leverage language transfer for three low-resource Creole languages, which exhibit relatedness to different language groups across distinct linguistic dimensions. Our approach improves performance substantially over baselines. However, we find that linguistic relatedness -- or even a lack thereof -- does not covary meaningfully with adapter performance. Surprisingly, our cross-attention fine-tuning approach appears equally effective with randomly initialized adapters, implying that the benefit of adapters in this setting lies in parameter regularization, and not in meaningful information transfer. We provide analysis supporting this regularization hypothesis. Our findings underscore the reality that neural language processing involves many success factors, and that not all neural methods leverage linguistic knowledge in intuitive ways.