Zero-Shot Cross-Lingual Sentiment Classification under Distribution Shift: an Exploratory Study

TL;DR

This study investigates zero-shot cross-lingual sentiment classification under distribution shifts, analyzing the effects of language and domain changes, and proposes cost-effective methods to improve out-of-distribution generalization using large language models.

Contribution

It provides the first analysis of OOD generalization in multilingual models, evaluates the impact of counterfactual data, and introduces new LLM-based approaches that outperform CAD without costly annotations.

Findings

OOD performance declines with distribution shifts.

Counterfactuals from high-resource languages help low-resource languages.

Proposed LLM-based methods improve accuracy by up to 3.1%.

Abstract

The brittleness of finetuned language model performance on out-of-distribution (OOD) test samples in unseen domains has been well-studied for English, yet is unexplored for multi-lingual models. Therefore, we study generalization to OOD test data specifically in zero-shot cross-lingual transfer settings, analyzing performance impacts of both language and domain shifts between train and test data. We further assess the effectiveness of counterfactually augmented data (CAD) in improving OOD generalization for the cross-lingual setting, since CAD has been shown to benefit in a monolingual English setting. Finally, we propose two new approaches for OOD generalization that avoid the costly annotation process associated with CAD, by exploiting the power of recent large language models (LLMs). We experiment with 3 multilingual models, LaBSE, mBERT, and XLM-R trained on English IMDb movie reviews, and evaluate on OOD test sets in 13 languages: Amazon product reviews, Tweets, and Restaurant reviews. Results echo the OOD performance decline observed in the monolingual English setting. Further, (i) counterfactuals from the original high-resource language do improve OOD generalization in the low-resource language, and (ii) our newly proposed cost-effective approaches reach similar or up to +3.1% better accuracy than CAD for Amazon and Restaurant reviews.