TartuNLP @ SIGTYP 2024 Shared Task: Adapting XLM-RoBERTa for Ancient and Historical Languages

TL;DR

This paper describes a lightweight adapter-based approach to adapt XLM-RoBERTa for various NLP tasks in ancient and historical languages, demonstrating competitive performance across multiple languages.

Contribution

The paper introduces a uniform, parameter-efficient fine-tuning method using adapters for multiple NLP tasks in ancient and historical languages, achieving competitive results.

Findings

Second place overall in shared task

First place in word-level gap-filling

Feasibility of adapting modern language models to historical languages

Abstract

We present our submission to the unconstrained subtask of the SIGTYP 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages for morphological annotation, POS-tagging, lemmatization, character- and word-level gap-filling. We developed a simple, uniform, and computationally lightweight approach based on the adapters framework using parameter-efficient fine-tuning. We applied the same adapter-based approach uniformly to all tasks and 16 languages by fine-tuning stacked language- and task-specific adapters. Our submission obtained an overall second place out of three submissions, with the first place in word-level gap-filling. Our results show the feasibility of adapting language models pre-trained on modern languages to historical and ancient languages via adapter training.