Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformers

TL;DR

This paper presents a novel approach using character-aware hierarchical transformers and character-level T5 models to improve NLP tasks like PoS tagging, morphological tagging, and lemmatization for low-resource historical languages, achieving top performance in a shared task.

Contribution

It introduces a hierarchical tokenization method combined with DeBERTa-V3 and character-level T5 models for low-resource language analysis, demonstrating state-of-the-art results.

Findings

Achieved first place in the constrained subtask of SIGTYP 2024.

Models nearly matched the performance of unconstrained top models.

Effective use of character-aware hierarchical transformers for low-resource NLP tasks.

Abstract

Historical languages present unique challenges to the NLP community, with one prominent hurdle being the limited resources available in their closed corpora. This work describes our submission to the constrained subtask of the SIGTYP 2024 shared task, focusing on PoS tagging, morphological tagging, and lemmatization for 13 historical languages. For PoS and morphological tagging we adapt a hierarchical tokenization method from Sun et al. (2023) and combine it with the advantages of the DeBERTa-V3 architecture, enabling our models to efficiently learn from every character in the training data. We also demonstrate the effectiveness of character-level T5 models on the lemmatization task. Pre-trained from scratch with limited data, our models achieved first place in the constrained subtask, nearly reaching the performance levels of the unconstrained task's winner. Our code is available at https://github.com/bowphs/SIGTYP-2024-hierarchical-transformers