HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samples

TL;DR

This paper introduces HinTel-AlignBench, a comprehensive benchmark and framework for evaluating vision-language models on Hindi and Telugu, addressing evaluation gaps and analyzing model performance across diverse Indian language datasets.

Contribution

It presents a semi-automated dataset creation framework, the HinTel-AlignBench benchmark with diverse datasets, and a detailed analysis of SOTA model performance on Indian languages.

Findings

Models perform worse on Indian languages compared to English.

Average performance regression is 8.3 points in Hindi and 5.5 in Telugu.

Identifies common failure modes to guide future improvements.

Abstract

With nearly 1.5 billion people and more than 120 major languages, India represents one of the most diverse regions in the world. As multilingual Vision-Language Models (VLMs) gain prominence, robust evaluation methodologies are essential to drive progress toward equitable AI for low-resource languages. Current multilingual VLM evaluations suffer from four major limitations: reliance on unverified auto-translations, narrow task/domain coverage, limited sample sizes, and lack of cultural and natively sourced Question-Answering (QA). To address these gaps, we present a scalable framework to evaluate VLMs in Indian languages and compare it with performance in English. Using the framework, we generate HinTel-AlignBench, a benchmark that draws from diverse sources in Hindi and Telugu with English-aligned samples. Our contributions are threefold: (1) a semi-automated dataset creation framework combining back-translation, filtering, and human verification; (2) the most comprehensive vision-language benchmark for Hindi and and Telugu, including adapted English datasets (VQAv2, RealWorldQA, CLEVR-Math) and native novel Indic datasets (JEE for STEM, VAANI for cultural grounding) with approximately 4,000 QA pairs per language; and (3) a detailed performance analysis of various State-of-the-Art (SOTA) open-weight and closed-source VLMs. We find a regression in performance for tasks in English versus in Indian languages for 4 out of 5 tasks across all the models, with an average regression of 8.3 points in Hindi and 5.5 points for Telugu. We categorize common failure modes to highlight concrete areas of improvement in multilingual multimodal understanding.