Enhancing BERT Fine-Tuning for Sentiment Analysis in Lower-Resourced Languages

TL;DR

This paper introduces a novel fine-tuning pipeline combining Active Learning, clustering, and dynamic data selection to improve sentiment analysis in low-resource languages, achieving significant annotation savings and performance gains.

Contribution

It proposes an integrated fine-tuning approach using AL schedulers and clustering, specifically tailored for low-resource language models, which is a new methodology in this context.

Findings

Up to 30% annotation savings achieved.

Performance improvements of up to four F1 score points.

Enhanced fine-tuning stability in low-resource language models.

Abstract

Limited data for low-resource languages typically yield weaker language models (LMs). Since pre-training is compute-intensive, it is more pragmatic to target improvements during fine-tuning. In this work, we examine the use of Active Learning (AL) methods augmented by structured data selection strategies which we term 'Active Learning schedulers', to boost the fine-tuning process with a limited amount of training data. We connect the AL to data clustering and propose an integrated fine-tuning pipeline that systematically combines AL, clustering, and dynamic data selection schedulers to enhance model's performance. Experiments in the Slovak, Maltese, Icelandic and Turkish languages show that the use of clustering during the fine-tuning phase together with AL scheduling can simultaneously produce annotation savings up to 30% and performance improvements up to four F1 score points, while also providing better fine-tuning stability.