DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer

TL;DR

DeFT-X introduces a denoising step using singular value decomposition before sparse fine-tuning, significantly improving zero-shot cross-lingual transfer performance for low-resource languages.

Contribution

It proposes a novel denoised sparse fine-tuning method that enhances robustness and effectiveness in cross-lingual transfer tasks.

Findings

DeFT-X outperforms existing SFT methods on low-resource language benchmarks.

Denoising improves the robustness of sparse masks in cross-lingual transfer.

DeFT-X achieves comparable or better results than state-of-the-art baselines.

Abstract

Effective cross-lingual transfer remains a critical challenge in scaling the benefits of large language models from high-resource to low-resource languages. Towards this goal, prior studies have explored many approaches to combine task knowledge from task-specific data in a (high-resource) source language and language knowledge from unlabeled text in a (low-resource) target language. One notable approach proposed composable sparse fine-tuning (SFT) for cross-lingual transfer that learns task-specific and language-specific sparse masks to select a subset of the pretrained model's parameters that are further fine-tuned. These sparse fine-tuned vectors (SFTs) are subsequently composed with the pretrained model to facilitate zero-shot cross-lingual transfer to a task in a target language, using only task-specific data from a source language. These sparse masks for SFTs were identified using a simple magnitude-based pruning. In our work, we introduce DeFT-X, a novel composable SFT approach that denoises the weight matrices of a pretrained model before magnitude pruning using singular value decomposition, thus yielding more robust SFTs. We evaluate DeFT-X on a diverse set of extremely low-resource languages for sentiment classification (NusaX) and natural language inference (AmericasNLI) and demonstrate that it performs at par or outperforms SFT and other prominent cross-lingual transfer baselines.