GanitBench: A bi-lingual benchmark for evaluating mathematical reasoning in Vision Language Models

TL;DR

GanitBench is a bilingual benchmark with 1527 math questions in English and Hindi, designed to evaluate vision-language models' reasoning abilities across languages and topics, highlighting current model limitations.

Contribution

Introduces GanitBench, a novel bilingual math reasoning benchmark in English and Hindi, with diverse question formats and evaluation of state-of-the-art models' performance and language biases.

Findings

GPT-4o mini outperforms other models with 38.15% accuracy.

Two-shot Chain-of-Thought improves model performance.

Model accuracy decreases when answering questions in Hindi.

Abstract

Benchmarks for evaluating reasoning among Vision Language Models (VLMs) on several fields and domains are being curated more frequently over the last few years. However these are often monolingual, mostly available in English. Additionally there also is a lack of datasets available in Hindi on tasks apart from comprehension and translation. We introduce GanitBench, a tough benchmark consisting of 1527 vision-only questions covering several topics in Mathematics - available in languages English and Hindi. Collected from two major examinations from India, the JEE Advanced and the CBSE Boards examinations, this benchmark includes questions in the form of images comprising of figures essential to a question as well as text. We evaluate two closed source models for the same, in zero-shot Chain-of-Thought (CoT) and two-shot CoT settings. GPT-4o mini is found to be the more dominant model on the benchmark, with it's highest average accuracy being 38.15%. We also evaluate models through a "Double Lock" constraint, which brings down the performance of the models by considerable margins. We observe that two-shot CoT appears to be a more effective setting under this environment. Performance of the two VLMs also decreases when answering the same questions in the Hindi language. We hope to facilitate the inclusion of languages like Hindi in research through our work.