GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models

TL;DR

GRAB is a new challenging benchmark designed to evaluate large multimodal models' graph analysis capabilities, highlighting current limitations and guiding future advancements in the field.

Contribution

Introduces GRAB, a synthetic, comprehensive graph analysis benchmark with 3284 questions across multiple tasks, to evaluate and push the limits of large multimodal models.

Findings

Current LMMs perform poorly on GRAB, with top score only 21%.

GRAB reveals specific areas where models struggle in graph reasoning.

Benchmark and lightweight version released to foster progress.

Abstract

Large multimodal models (LMMs) have exhibited proficiencies across many visual tasks. Although numerous well-known benchmarks exist to evaluate model performance, they increasingly have insufficient headroom. As such, there is a pressing need for a new generation of benchmarks challenging enough for the next generation of LMMs. One area that LMMs show potential is graph analysis, specifically, the tasks an analyst might typically perform when interpreting figures such as estimating the mean, intercepts or correlations of functions and data series. In this work, we introduce GRAB, a graph analysis benchmark, fit for current and future frontier LMMs. Our benchmark is predominantly synthetic, ensuring high-quality, noise-free questions. GRAB is comprised of 3284 questions, covering five tasks and 23 graph properties. We evaluate 20 LMMs on GRAB, finding it to be a challenging benchmark, with the highest performing model attaining a score of just 21.0%. Finally, we conduct various ablations to investigate where the models succeed and struggle. We release GRAB and a lightweight GRAB-Lite to encourage progress in this important, growing domain.