Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems

TL;DR

This paper evaluates the reasoning capabilities of multimodal large language models on Bongard Problems, revealing their limitations in synthetic visual reasoning despite some success on real-world datasets.

Contribution

It introduces Bongard-RWR, a new dataset with real-world images for synthetic BPs, and systematically analyzes MLLMs' reasoning limitations across diverse datasets.

Findings

MLLMs perform poorly on synthetic Bongard Problems

Weak performance is due to general visual reasoning limitations

Some success observed on real-world datasets

Abstract

Abstract visual reasoning (AVR) involves discovering shared concepts across images through analogy, akin to solving IQ test problems. Bongard Problems (BPs) remain a key challenge in AVR, requiring both visual reasoning and verbal description. We investigate whether multimodal large language models (MLLMs) can solve BPs by formulating a set of diverse MLLM-suited solution strategies and testing $4$ proprietary and $4$ open-access models on $3$ BP datasets featuring synthetic (classic BPs) and real-world (Bongard HOI and Bongard-OpenWorld) images. Despite some successes on real-world datasets, MLLMs struggle with synthetic BPs. To explore this gap, we introduce Bongard-RWR, a dataset representing synthetic BP concepts using real-world images. Our findings suggest that weak MLLM performance on classical BPs is not due to the domain specificity, but rather comes from their general AVR limitations. Code and dataset are available at: https://github.com/pavonism/bongard-rwr