Donors and Recipients: On Asymmetric Transfer Across Tasks and Languages with Parameter-Efficient Fine-Tuning

TL;DR

This study investigates how parameter-efficient fine-tuning of large language models transfers improvements across different tasks and languages, revealing asymmetric transfer patterns and the importance of language identity.

Contribution

It systematically analyzes transfer effects across multiple models, tasks, and languages, highlighting the dominance of matched-task cross-language transfer and the role of language resource levels.

Findings

Matched-Task (Cross-Language) transfer is most effective.

High-resource languages act as efficient donors.

Specialized tasks and low-resource languages are more isolated.

Abstract

Large language models (LLMs) perform strongly across tasks and languages, yet how improvements in one task or language affect other tasks and languages remains poorly understood. We conduct a controlled LoRA fine-tuning study across multiple open-weight LLM families and scales, using a standardised grid of 11 languages and four benchmarks. We fine-tune each model on a single task-language source and measure transfer when evaluated on all other task-language target pairs. We decompose transfer into three regimes: (i) Matched-Task (Cross-Language), (ii) Matched-Language (Cross-Task), and (iii) Cross-Task (Cross-Language). Single-source fine-tuning yields a net positive uplift across regimes, but the gains are strongly asymmetric. Matched-Task (Cross-Language) transfer emerges as the most effective and predictable regime, driven principally by the identity of the target language rather than model architecture. We identify a stable hierarchy where high-resource languages and broad semantic tasks act as efficient recipients that absorb gains from diverse sources, while specialised tasks and lower-resource languages are more isolated. These results imply that effective fine-tuning requires navigating donor-recipient roles to maximise downstream gains.