From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars

TL;DR

This paper introduces benchmarks and a retrieval-augmented approach to evaluate and improve language models' ability to interpret complex linguistic descriptions for low-resource languages, enabling better NLP tasks like translation.

Contribution

It presents the first comprehensive benchmarks and a novel retrieval-augmented method for extracting linguistic features from formal grammar descriptions.

Findings

Benchmark covers 248 languages and 142 families.

RAG-based approach improves feature extraction accuracy.

Provides publicly available code and data for further research.

Abstract

Recent advances in language modeling have demonstrated significant improvements in zero-shot capabilities, including in-context learning, instruction following, and machine translation for extremely under-resourced languages (Tanzer et al., 2024). However, many languages with limited written resources rely primarily on formal descriptions of grammar and vocabulary. In this paper, we introduce a set of benchmarks to evaluate how well models can extract and classify information from the complex descriptions found in linguistic grammars. We present a Retrieval-Augmented Generation (RAG)-based approach that leverages these descriptions for downstream tasks such as machine translation. Our benchmarks encompass linguistic descriptions for 248 languages across 142 language families, focusing on typological features from WALS and Grambank. This set of benchmarks offers the first comprehensive evaluation of language models' in-context ability to accurately interpret and extract linguistic features, providing a critical resource for scaling NLP to low-resource languages. The code and data are publicly available at \url{https://github.com/al-the-eigenvalue/RAG-on-grammars}.