SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture

TL;DR

SANSKRITI is a large, detailed benchmark dataset designed to evaluate how well language models understand Indian culture across various regions and attributes, highlighting disparities in cultural knowledge among different models.

Contribution

Introduces the largest culturally diverse benchmark for Indian culture, enabling comprehensive evaluation of language models' cultural understanding.

Findings

Significant disparities in model performance on region-specific questions

Many models struggle with culturally nuanced queries

SANSKRITI sets a new standard for cultural evaluation in LMs

Abstract

Language Models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce SANSKRITI, a benchmark designed to evaluate language models' comprehension of India's rich cultural diversity. Comprising 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, SANSKRITI is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture: rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife, and personalities, providing a comprehensive representation of India's cultural tapestry. We evaluate SANSKRITI on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, SANSKRITI sets a new standard for assessing and improving the cultural understanding of LMs.