MATA: Mindful Assessment of the Telugu Abilities of Large Language Models

TL;DR

This paper introduces MATA, a comprehensive Telugu language evaluation dataset for LLMs, revealing their reliance on superficial heuristics and comparing model judgments with human assessments to improve linguistic capabilities.

Contribution

The paper presents MATA, a new Telugu language dataset for LLM evaluation, and analyzes model performance, heuristics reliance, and evaluation reliability in low-resource settings.

Findings

LLMs rely on superficial heuristics like answer position.

Performance varies significantly across models.

Model evaluation correlates with human judgment in open-ended tasks.

Abstract

In this paper, we introduce MATA, a novel evaluation dataset to assess the ability of Large Language Models (LLMs) in Telugu language, comprising 729 carefully curated multiple-choice and open-ended questions that span diverse linguistic dimensions. We evaluate 11 open-weight and closed-source LLMs on our dataset and present a fine-grained analysis of their performance. Further, we empirically show how LLMs rely on superficial heuristics such as answer position and distractor patterns for multiple-choice questions. Finally, we also compare LLM-as-a-judge evaluation with human evaluation for open-ended questions assess its reliability in a low-resource language. We argue that such fine-grained evaluation is essential for understanding model limitations and can inform the development of more linguistically capable LLMs, while also serving as a foundation for future research in Telugu NLP. Our dataset is available at: https://huggingface.co/datasets/TeluguLLMResearch/MATA