NativQA: Multilingual Culturally-Aligned Natural Query for LLMs

TL;DR

NativQA introduces a scalable framework for creating culturally and regionally aligned multilingual QA datasets, enabling better evaluation and fine-tuning of LLMs across diverse languages and regions.

Contribution

The paper presents a novel, language-independent framework for constructing large-scale, region-specific QA datasets in native languages for LLM evaluation and development.

Findings

Created MultiNativQA with ~64k QA pairs in 7 languages

Benchmarking shows variability in LLM performance across languages

Framework is effective for low-resource language datasets

Abstract

Natural Question Answering (QA) datasets play a crucial role in evaluating the capabilities of large language models (LLMs), ensuring their effectiveness in real-world applications. Despite the numerous QA datasets that have been developed and some work has been done in parallel, there is a notable lack of a framework and large scale region-specific datasets queried by native users in their own languages. This gap hinders the effective benchmarking and the development of fine-tuned models for regional and cultural specificities. In this study, we propose a scalable, language-independent framework, NativQA, to seamlessly construct culturally and regionally aligned QA datasets in native languages, for LLM evaluation and tuning. We demonstrate the efficacy of the proposed framework by designing a multilingual natural QA dataset, MultiNativQA, consisting of ~64k manually annotated QA pairs in seven languages, ranging from high to extremely low resource, based on queries from native speakers from 9 regions covering 18 topics. We benchmark open- and closed-source LLMs with the MultiNativQA dataset. We made the MultiNativQA dataset(https://huggingface.co/datasets/QCRI/MultiNativQA), and other experimental scripts(https://gitlab.com/nativqa/multinativqa) publicly available for the community.