PEFTT: Parameter-Efficient Fine-Tuning for low-resource Tibetan pre-trained language models

TL;DR

This paper explores efficient fine-tuning methods for low-resource Tibetan language models, demonstrating significant improvements and addressing a crucial gap in Tibetan NLP for large language models.

Contribution

It introduces and evaluates prompt-tuning, Adapter fine-tuning, and their combination for Tibetan, a low-resource language, filling a research gap in efficient LLM adaptation.

Findings

Prompt-tuning improves performance significantly.

Adapter fine-tuning enhances model efficiency.

Combined methods yield the best results.

Abstract

In this era of large language models (LLMs), the traditional training of models has become increasingly unimaginable for regular users and institutions. The exploration of efficient fine-tuning for high-resource languages on these models is an undeniable trend that is gradually gaining popularity. However, there has been very little exploration for various low-resource languages, such as Tibetan. Research in Tibetan NLP is inherently scarce and limited. While there is currently no existing large language model for Tibetan due to its low-resource nature, that day will undoubtedly arrive. Therefore, research on efficient fine-tuning for low-resource language models like Tibetan is highly necessary. Our research can serve as a reference to fill this crucial gap. Efficient fine-tuning strategies for pre-trained language models (PLMs) in Tibetan have seen minimal exploration. We conducted three types of efficient fine-tuning experiments on the publicly available TNCC-title dataset: "prompt-tuning," "Adapter lightweight fine-tuning," and "prompt-tuning + Adapter fine-tuning." The experimental results demonstrate significant improvements using these methods, providing valuable insights for advancing Tibetan language applications in the context of pre-trained models.