All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark

TL;DR

The paper presents MAIA, a comprehensive Italian benchmark for evaluating multimodal reasoning in visual language models on videos, focusing on reasoning categories, cultural relevance, and language-specific challenges.

Contribution

It introduces MAIA, a novel Italian video benchmark with detailed reasoning categories and dual tasks, to assess and analyze the reasoning capabilities of vision-language models.

Findings

Models show low performance, indicating fragility in reasoning abilities.

MAIA effectively disentangles language and vision relations.

The benchmark highlights the need for improved multimodal reasoning models.

Abstract

We introduce MAIA (Multimodal AI Assessment), a native-Italian benchmark designed for fine-grained investigation of the reasoning abilities of visual language models on videos. MAIA differs from other available video benchmarks for its design, its reasoning categories, the metric it uses, and the language and culture of the videos. MAIA evaluates Vision Language Models (VLMs) on two aligned tasks: a visual statement verification task, and an open-ended visual question-answering task, both on the same set of video-related questions. It considers twelve reasoning categories that aim to disentangle language and vision relations by highlighting the role of the visual input. Thanks to its carefully taught design, it evaluates VLMs' consistency and visually grounded natural language comprehension and generation simultaneously through an aggregated metric revealing low results that highlight models' fragility. Last but not least, the video collection has been carefully selected to reflect the Italian culture, and the language data are produced by native-speakers.