PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models

TL;DR

PISA-Bench is a multilingual, multimodal benchmark based on PISA tests, designed to evaluate vision-language models across six languages and various reasoning tasks, highlighting current models' limitations especially in non-English and complex reasoning scenarios.

Contribution

This paper introduces PISA-Bench, a high-quality, multilingual, and multimodal benchmark derived from PISA assessments, addressing limitations of existing datasets and enabling comprehensive evaluation of vision-language models.

Findings

Small models (<20B parameters) underperform on PISA-Bench.

Models show significant performance drops on non-English languages.

High error rates in spatial and geometric reasoning tasks.

Abstract

Vision-language models (VLMs) have demonstrated remarkable progress in multimodal reasoning. However, existing benchmarks remain limited in terms of high-quality, human-verified examples. Many current datasets rely on synthetically generated content by large language models (LLMs). Furthermore, most datasets are limited to English, as manual quality assurance of translated samples is time-consuming and costly. To fill this gap, we introduce PISA-Bench, a multilingual benchmark derived from English examples of the expert-created PISA tests, a unified framework for the assessment of student competencies in over eighty countries. Each example consists of human-extracted instructions, questions, answer options, and images, enriched with question type categories, and has been translated from English into five additional languages (Spanish, German, Chinese, French, and Italian), resulting in a fully parallel corpus covering six languages. We evaluate state-of-the-art vision-language models on PISA-Bench and find that especially small models (<20B parameters) fail to achieve high test scores. We further find substantial performance degradation on non-English splits as well as high error-rates when models are tasked with spatial and geometric reasoning. By releasing the dataset and evaluation framework, we provide a resource for advancing research on multilingual multimodal reasoning.