It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs
Yue Li, Zhixue Zhao, Carolina Scarton

TL;DR
This paper investigates whether large language models can learn extremely low-resource languages through in-context learning, comparing it to fine-tuning, and provides practical guidelines for practitioners working with such languages.
Contribution
It offers the first comprehensive analysis of ICL for extremely low-resource languages and compares its effectiveness to PEFT, highlighting practical adaptation strategies.
Findings
Zero-shot ICL with language alignment is highly effective for extremely low-resource languages.
PEFT performs poorly when both language and script are under-represented.
Few-shot ICL or PEFT benefits languages with better representation in LLMs.
Abstract
Extremely low-resource languages, especially those written in rare scripts, as shown in Figure 1, remain largely unsupported by large language models (LLMs). This is due in part to compounding factors such as the lack of training data. This paper delivers the first comprehensive analysis of whether LLMs can acquire such languages purely via in-context learning (ICL), with or without auxiliary alignment signals, and how these methods compare to parameter-efficient fine-tuning (PEFT). We systematically evaluate 20 under-represented languages across three state-of-the-art multilingual LLMs. Our findings highlight the limitation of PEFT when both language and its script are extremely under-represented by the LLM. In contrast, zero-shot ICL with language alignment is impressively effective on extremely low-resource languages, while few-shot ICL or PEFT is more beneficial for languages…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
