A cost-benefit analysis of cross-lingual transfer methods

TL;DR

This paper evaluates various cross-lingual transfer methods by analyzing their effectiveness, costs, and latencies, demonstrating that combining zero-shot and translation approaches can achieve state-of-the-art results and questioning the necessity of target language training data.

Contribution

It provides a comprehensive cost-benefit analysis of cross-lingual transfer methods, highlighting task-dependent effectiveness and proposing combined approaches for improved performance.

Findings

Best method varies by task

Combining zero-shot and translation yields state-of-the-art results

Manual target language data may be unnecessary

Abstract

An effective method for cross-lingual transfer is to fine-tune a bilingual or multilingual model on a supervised dataset in one language and evaluating it on another language in a zero-shot manner. Translating examples at training time or inference time are also viable alternatives. However, there are costs associated with these methods that are rarely addressed in the literature. In this work, we analyze cross-lingual methods in terms of their effectiveness (e.g., accuracy), development and deployment costs, as well as their latencies at inference time. Our experiments on three tasks indicate that the best cross-lingual method is highly task-dependent. Finally, by combining zero-shot and translation methods, we achieve the state-of-the-art in two of the three datasets used in this work. Based on these results, we question the need for manually labeled training data in a target language. Code and translated datasets are available at https://github.com/unicamp-dl/cross-lingual-analysis