KinyaColBERT: A Lexically Grounded Retrieval Model for Low-Resource Retrieval-Augmented Generation

TL;DR

KinyaColBERT is a novel retrieval model designed for low-resource languages, combining morphology-based tokenization and late word-level interactions to improve retrieval accuracy in retrieval-augmented generation systems.

Contribution

The paper introduces KinyaColBERT, a new retrieval model that enhances low-resource language retrieval through morphology-based tokenization and late interaction mechanisms.

Findings

KinyaColBERT outperforms existing baselines on Kinyarwanda retrieval tasks.

Morphology-based tokenization improves language coverage and retrieval accuracy.

The model offers a cost-effective solution for low-resource RAG applications.

Abstract

The recent mainstream adoption of large language model (LLM) technology is enabling novel applications in the form of chatbots and virtual assistants across many domains. With the aim of grounding LLMs in trusted domains and avoiding the problem of hallucinations, retrieval-augmented generation (RAG) has emerged as a viable solution. In order to deploy sustainable RAG systems in low-resource settings, achieving high retrieval accuracy is not only a usability requirement but also a cost-saving strategy. Through empirical evaluations on a Kinyarwanda-language dataset, we find that the most limiting factors in achieving high retrieval accuracy are limited language coverage and inadequate sub-word tokenization in pre-trained language models. We propose a new retriever model, KinyaColBERT, which integrates two key concepts: late word-level interactions between queries and documents, and a morphology-based tokenization coupled with two-tier transformer encoding. This methodology results in lexically grounded contextual embeddings that are both fine-grained and self-contained. Our evaluation results indicate that KinyaColBERT outperforms strong baselines and leading commercial text embedding APIs on a Kinyarwanda agricultural retrieval benchmark. By adopting this retrieval strategy, we believe that practitioners in other low-resource settings can not only achieve reliable RAG systems but also deploy solutions that are more cost-effective.