Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Models

TL;DR

This paper introduces a new benchmark and evaluation methods for assessing the predictive reasoning capabilities of multimodal large language models across various scenarios, highlighting their strengths and weaknesses.

Contribution

It presents a novel benchmark and evaluation framework specifically designed to measure predictive reasoning in multimodal large language models, filling a significant research gap.

Findings

Current MLLMs show varied performance in predictive reasoning tasks.

The benchmark effectively differentiates model capabilities in diverse scenarios.

Evaluation methods provide robust quantification of future prediction and reasoning abilities.

Abstract

Multimodal large language models (MLLMs) have shown great potential in perception and interpretation tasks, but their capabilities in predictive reasoning remain under-explored. To address this gap, we introduce a novel benchmark that assesses the predictive reasoning capabilities of MLLMs across diverse scenarios. Our benchmark targets three important domains: abstract pattern reasoning, human activity prediction, and physical interaction prediction. We further develop three evaluation methods powered by large language model to robustly quantify a model's performance in predicting and reasoning the future based on multi-visual context. Empirical experiments confirm the soundness of the proposed benchmark and evaluation methods via rigorous testing and reveal pros and cons of current popular MLLMs in the task of predictive reasoning. Lastly, our proposed benchmark provides a standardized evaluation framework for MLLMs and can facilitate the development of more advanced models that can reason and predict over complex long sequence of multimodal input.