Transformers on Multilingual Clause-Level Morphology

TL;DR

This paper presents transformer-based systems for multilingual clause-level morphology tasks, achieving first place in inflection, reinflection, and analysis by combining data augmentation and prefix-tuning techniques.

Contribution

It introduces effective transformer approaches, including data augmentation and prefix-tuning, for multilingual morphological tasks, setting new state-of-the-art results in the shared task.

Findings

Data augmentation improves performance in inflection and reinflection.

Prefix-tuning on mGPT enhances analysis in low-data and multilingual settings.

Transformers with data augmentation outperform other methods in key tasks.

Abstract

This paper describes our winning systems in MRL: The 1st Shared Task on Multilingual Clause-level Morphology (EMNLP 2022 Workshop) designed by KUIS AI NLP team. We present our work for all three parts of the shared task: inflection, reinflection, and analysis. We mainly explore transformers with two approaches: (i) training models from scratch in combination with data augmentation, and (ii) transfer learning with prefix-tuning at multilingual morphological tasks. Data augmentation significantly improves performance for most languages in the inflection and reinflection tasks. On the other hand, Prefix-tuning on a pre-trained mGPT model helps us to adapt analysis tasks in low-data and multilingual settings. While transformer architectures with data augmentation achieved the most promising results for inflection and reinflection tasks, prefix-tuning on mGPT received the highest results for the analysis task. Our systems received 1st place in all three tasks in MRL 2022.