Composable Sparse Fine-Tuning for Cross-Lingual Transfer

TL;DR

This paper introduces a novel sparse fine-tuning method that combines the benefits of adapters and sparse control, enabling efficient, composable, and effective cross-lingual transfer without increasing model size.

Contribution

It proposes a new sparse, composable fine-tuning technique based on the Lottery Ticket Hypothesis, outperforming adapters in zero-shot cross-lingual transfer without adding parameters.

Findings

Outperforms adapters in zero-shot cross-lingual transfer benchmarks

Sparsity prevents interference and overfitting in composable fine-tuning

Method does not increase parameters or alter model architecture

Abstract

Fine-tuning the entire set of parameters of a large pretrained model has become the mainstream approach for transfer learning. To increase its efficiency and prevent catastrophic forgetting and interference, techniques like adapters and sparse fine-tuning have been developed. Adapters are modular, as they can be combined to adapt a model towards different facets of knowledge (e.g., dedicated language and/or task adapters). Sparse fine-tuning is expressive, as it controls the behavior of all model components. In this work, we introduce a new fine-tuning method with both these desirable properties. In particular, we learn sparse, real-valued masks based on a simple variant of the Lottery Ticket Hypothesis. Task-specific masks are obtained from annotated data in a source language, and language-specific masks from masked language modeling in a target language. Both these masks can then be composed with the pretrained model. Unlike adapter-based fine-tuning, this method neither increases the number of parameters at inference time nor alters the original model architecture. Most importantly, it outperforms adapters in zero-shot cross-lingual transfer by a large margin in a series of multilingual benchmarks, including Universal Dependencies, MasakhaNER, and AmericasNLI. Based on an in-depth analysis, we additionally find that sparsity is crucial to prevent both 1) interference between the fine-tunings to be composed and 2) overfitting. We release the code and models at https://github.com/cambridgeltl/composable-sft.