Boosting the Capabilities of Compact Models in Low-Data Contexts with Large Language Models and Retrieval-Augmented Generation

TL;DR

This paper introduces a retrieval-augmented generation framework using large language models to enhance small models for morphological glossing in low-resource languages, significantly improving performance and interpretability.

Contribution

It presents a novel RAG-based approach that combines linguistic grammars, LLM interpretive power, and small models to excel in low-data morphological tasks.

Findings

Achieved state-of-the-art results in morphological glossing for low-resource languages.

Demonstrated significant performance improvements with linguistic inputs and LLM support.

Provided more reliable and explainable outputs for documentary linguists.

Abstract

The data and compute requirements of current language modeling technology pose challenges for the processing and analysis of low-resource languages. Declarative linguistic knowledge has the potential to partially bridge this data scarcity gap by providing models with useful inductive bias in the form of language-specific rules. In this paper, we propose a retrieval augmented generation (RAG) framework backed by a large language model (LLM) to correct the output of a smaller model for the linguistic task of morphological glossing. We leverage linguistic information to make up for the lack of data and trainable parameters, while allowing for inputs from written descriptive grammars interpreted and distilled through an LLM. The results demonstrate that significant leaps in performance and efficiency are possible with the right combination of: a) linguistic inputs in the form of grammars, b) the interpretive power of LLMs, and c) the trainability of smaller token classification networks. We show that a compact, RAG-supported model is highly effective in data-scarce settings, achieving a new state-of-the-art for this task and our target languages. Our work also offers documentary linguists a more reliable and more usable tool for morphological glossing by providing well-reasoned explanations and confidence scores for each output.