NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models

TL;DR

NTSEBench is a new multi-modal reasoning dataset with 2728 questions and 4642 images, designed to evaluate the cognitive reasoning abilities of large vision-language models beyond simple pattern recognition.

Contribution

The paper introduces NTSEBench, a comprehensive dataset for assessing complex cognitive reasoning in vision-language models, along with baseline evaluations and modeling strategies.

Findings

State-of-the-art models show limited performance on complex reasoning tasks.

The dataset covers diverse question types from the NTSE exam.

Proposed strategies enable better multi-modal reasoning handling.

Abstract

Cognitive textual and visual reasoning tasks, including puzzles, series, and analogies, demand the ability to quickly reason, decipher, and evaluate patterns both textually and spatially. Due to extensive training on vast amounts of human-curated data, LLMs and VLMs excel in common-sense reasoning tasks, however still struggle with more complex reasoning that demands deeper cognitive understanding. We introduce NTSEBench, a new dataset designed to evaluate cognitive multi-modal reasoning and problem-solving skills of large models. The dataset contains 2728 multiple-choice questions, accompanied by a total of 4,642 images, categorized into 26 different types. These questions are drawn from the nationwide NTSE examination in India and feature a mix of visual and textual general aptitude challenges, designed to assess intelligence and critical thinking skills beyond mere rote learning. We establish baselines on the dataset using state-of-the-art LLMs and VLMs. To facilitate a comparison between open source and propriety models, we propose four distinct modeling strategies to handle different modalities -- text and images -- in the dataset instances.