PrOnto: Language Model Evaluations for 859 Languages

TL;DR

PrOnto introduces an automated method to create evaluation datasets for hundreds of languages using New Testament translations, enabling assessment of pretrained language models without manual annotation.

Contribution

The paper presents a novel, annotation-free approach to generate language evaluation datasets from New Testament translations, covering 859 languages.

Findings

Effective dataset creation for 859 languages.

Demonstrated method accurately assesses language model quality.

Publicly released datasets for broad language evaluation.

Abstract

Evaluation datasets are critical resources for measuring the quality of pretrained language models. However, due to the high cost of dataset annotation, these resources are scarce for most languages other than English, making it difficult to assess the quality of language models. In this work, we present a new method for evaluation dataset construction which enables any language with a New Testament translation to receive a suite of evaluation datasets suitable for pretrained language model evaluation. The method critically involves aligning verses with those in the New Testament portion of English OntoNotes, and then projecting annotations from English to the target language, with no manual annotation required. We apply this method to 1051 New Testament translations in 859 and make them publicly available. Additionally, we conduct experiments which demonstrate the efficacy of our method for creating evaluation tasks which can assess language model quality.