Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction

TL;DR

This paper introduces the Cognate Transformer, a novel model adapted from protein language models, which automates phonological reconstruction and cognate reflex prediction, outperforming existing methods especially with pre-training.

Contribution

The paper presents a new application of the MSA Transformer for historical linguistics, specifically for phonological reconstruction and reflex prediction, demonstrating improved accuracy over prior models.

Findings

Outperforms existing models on both tasks

Pre-training on masked word prediction enhances performance

Effective adaptation of protein language models to linguistics

Abstract

Phonological reconstruction is one of the central problems in historical linguistics where a proto-word of an ancestral language is determined from the observed cognate words of daughter languages. Computational approaches to historical linguistics attempt to automate the task by learning models on available linguistic data. Several ideas and techniques drawn from computational biology have been successfully applied in the area of computational historical linguistics. Following these lines, we adapt MSA Transformer, a protein language model, to the problem of automated phonological reconstruction. MSA Transformer trains on multiple sequence alignments as input and is, thus, apt for application on aligned cognate words. We, hence, name our model as Cognate Transformer. We also apply the model on another associated task, namely, cognate reflex prediction, where a reflex word in a daughter language is predicted based on cognate words from other daughter languages. We show that our model outperforms the existing models on both tasks, especially when it is pre-trained on masked word prediction task.