Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages

TL;DR

This study evaluates how transfer learning and distant supervision enhance multilingual transformer models like mBERT and XLM-RoBERTa for low-resource African languages, demonstrating significant performance gains with minimal labeled data but also highlighting certain limitations.

Contribution

It provides an empirical analysis of low-resource African languages using transfer learning and distant supervision, revealing their effectiveness and challenges in NLP tasks.

Findings

Achieving comparable performance with as few as 10-100 labeled sentences.

Transfer learning and distant supervision significantly improve low-resource NLP tasks.

Identified scenarios where these methods do not perform as well, highlighting limitations.

Abstract

Multilingual transformer models like mBERT and XLM-RoBERTa have obtained great improvements for many NLP tasks on a variety of languages. However, recent works also showed that results from high-resource languages could not be easily transferred to realistic, low-resource scenarios. In this work, we study trends in performance for different amounts of available resources for the three African languages Hausa, isiXhosa and Yor\`ub\'a on both NER and topic classification. We show that in combination with transfer learning or distant supervision, these models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. However, we also find settings where this does not hold. Our discussions and additional experiments on assumptions such as time and hardware restrictions highlight challenges and opportunities in low-resource learning.