Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models

TL;DR

This paper introduces DZEN, a Dzongkha-English dataset for scientific questions, and evaluates large language models' performance in low-resource language question answering, highlighting challenges and strategies for improvement.

Contribution

The creation of a parallel Dzongkha-English question dataset and an analysis of LLM performance and prompting strategies in a low-resource language context.

Findings

LLMs perform significantly worse in Dzongkha than in English.

Chain-of-Thought prompting improves reasoning question accuracy.

Adding English translations enhances Dzongkha question response precision.

Abstract

In this work, we provide DZEN, a dataset of parallel Dzongkha and English test questions for Bhutanese middle and high school students. The over 5K questions in our collection span a variety of scientific topics and include factual, application, and reasoning-based questions. We use our parallel dataset to test a number of Large Language Models (LLMs) and find a significant performance difference between the models in English and Dzongkha. We also look at different prompting strategies and discover that Chain-of-Thought (CoT) prompting works well for reasoning questions but less well for factual ones. We also find that adding English translations enhances the precision of Dzongkha question responses. Our results point to exciting avenues for further study to improve LLM performance in Dzongkha and, more generally, in low-resource languages. We release the dataset at: https://github.com/kraritt/llm_dzongkha_evaluation.