Typologically Informed Parameter Aggregation

TL;DR

This paper introduces TIPA, a training-free method that creates proxy language adapters by aggregating existing adapters based on typological similarity, enabling effective zero-shot cross-lingual transfer across many languages.

Contribution

We propose TIPA, a novel typologically informed aggregation method that constructs proxy adapters without training, improving multilingual transfer especially for low-resource languages.

Findings

TIPA outperforms baselines like English-only fine-tuning.

TIPA achieves comparable results to dedicated language adapters.

Largest gains observed for languages without dedicated adapters.

Abstract

Massively multilingual language models enable cross-lingual generalization but underperform on low-resource and unseen languages. While adapter-based fine-tuning offers a parameter-efficient solution, training language-specific adapters at scale remains costly. We introduce Typologically Informed Parameter Aggregation (TIPA), a training-free method that constructs proxy language adapters by aggregating existing ones, weighted by typological similarity. Integrated into the MAD-X framework, these proxies enable zero-shot cross-lingual transfer without additional training. We evaluate TIPA on five NLP tasks and over 230 languages. TIPA consistently outperforms or matches baselines such as English-only fine-tuning or selecting the typologically closest language adapter. We see the largest gains for languages lacking dedicated adapters. Our results demonstrate that typologically informed aggregation provides a viable alternative to language-specific modules without any training needed.