INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

TL;DR

This paper introduces INCLUDE, a large, regional knowledge-focused benchmark with 197,243 QA pairs across 44 languages, to evaluate multilingual LLMs in real-world, culturally relevant contexts, addressing the lack of diverse evaluation resources.

Contribution

It creates a new multilingual benchmark based on regional exam data, emphasizing local knowledge and reasoning, to better assess LLMs' performance in diverse language environments.

Findings

Benchmark covers 44 languages with 197,243 QA pairs.

Evaluates LLMs' regional knowledge and reasoning capabilities.

Highlights performance gaps in multilingual models across different languages.

Abstract

The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.