The futility of STILTs for the classification of lexical borrowings in Spanish

TL;DR

This study evaluates the effectiveness of supplementary training on intermediate tasks (STILTs) for classifying lexical borrowings in Spanish, finding no significant improvements over direct fine-tuning of transformer models.

Contribution

The paper provides an empirical assessment of STILTs in the context of lexical borrowing detection in Spanish, demonstrating their limited benefit in this task.

Findings

STILTs do not improve classification performance over direct fine-tuning.

Multilingual models trained on small language subsets outperform multilingual BERT.

Multilingual RoBERTa performs better than multilingual BERT for this task.

Abstract

The first edition of the IberLEF 2021 shared task on automatic detection of borrowings (ADoBo) focused on detecting lexical borrowings that appeared in the Spanish press and that have recently been imported into the Spanish language. In this work, we tested supplementary training on intermediate labeled-data tasks (STILTs) from part of speech (POS), named entity recognition (NER), code-switching, and language identification approaches to the classification of borrowings at the token level using existing pre-trained transformer-based language models. Our extensive experimental results suggest that STILTs do not provide any improvement over direct fine-tuning of multilingual models. However, multilingual models trained on small subsets of languages perform reasonably better than multilingual BERT but not as good as multilingual RoBERTa for the given dataset.