Cross-lingual transfer of multilingual models on low resource African Languages

TL;DR

This paper evaluates the cross-lingual transfer of multilingual and monolingual models on low-resource African languages, demonstrating that multilingual models like AfriBERT can achieve high accuracy in transfer tasks with minimal resource requirements.

Contribution

It benchmarks the transfer performance of various models on Kinyarwanda and Kirundi, highlighting the effectiveness of multilingual models like AfriBERT in low-resource settings.

Findings

AfriBERT achieved 88.3% accuracy after fine-tuning.

BiGRU achieved 83.3% accuracy, outperforming other neural models.

Multilingual models show strong transfer capabilities in low-resource languages.

Abstract

Large multilingual models have significantly advanced natural language processing (NLP) research. However, their high resource demands and potential biases from diverse data sources have raised concerns about their effectiveness across low-resource languages. In contrast, monolingual models, trained on a single language, may better capture the nuances of the target language, potentially providing more accurate results. This study benchmarks the cross-lingual transfer capabilities from a high-resource language to a low-resource language for both, monolingual and multilingual models, focusing on Kinyarwanda and Kirundi, two Bantu languages. We evaluate the performance of transformer based architectures like Multilingual BERT (mBERT), AfriBERT, and BantuBERTa against neural-based architectures such as BiGRU, CNN, and char-CNN. The models were trained on Kinyarwanda and tested on Kirundi, with fine-tuning applied to assess the extent of performance improvement and catastrophic forgetting. AfriBERT achieved the highest cross-lingual accuracy of 88.3% after fine-tuning, while BiGRU emerged as the best-performing neural model with 83.3% accuracy. We also analyze the degree of forgetting in the original language post-fine-tuning. While monolingual models remain competitive, this study highlights that multilingual models offer strong cross-lingual transfer capabilities in resource limited settings.