Blocks as Probes: Dissecting Categorization Ability of Large Multimodal Models

TL;DR

This paper introduces ComBo, a new benchmark for evaluating the fundamental categorization abilities of large multimodal models, revealing their strengths and limitations in human-like perception and understanding.

Contribution

The paper presents ComBo, a novel benchmark that dissects the categorization process in LMMs, offering a detailed quantitative evaluation of their learning and usage capabilities.

Findings

LMMs show acceptable generalization in learning new categories

Gaps remain in fine-grained spatial perception and abstract understanding

Benchmark provides insights for improving interpretability and generalization

Abstract

Categorization, a core cognitive ability in humans that organizes objects based on common features, is essential to cognitive science as well as computer vision. To evaluate the categorization ability of visual AI models, various proxy tasks on recognition from datasets to open world scenarios have been proposed. Recent development of Large Multimodal Models (LMMs) has demonstrated impressive results in high-level visual tasks, such as visual question answering, video temporal reasoning, etc., utilizing the advanced architectures and large-scale multimodal instruction tuning. Previous researchers have developed holistic benchmarks to measure the high-level visual capability of LMMs, but there is still a lack of pure and in-depth quantitative evaluation of the most fundamental categorization ability. According to the research on human cognitive process, categorization can be seen as including two parts: category learning and category use. Inspired by this, we propose a novel, challenging, and efficient benchmark based on composite blocks, called ComBo, which provides a disentangled evaluation framework and covers the entire categorization process from learning to use. By analyzing the results of multiple evaluation tasks, we find that although LMMs exhibit acceptable generalization ability in learning new categories, there are still gaps compared to humans in many ways, such as fine-grained perception of spatial relationship and abstract category understanding. Through the study of categorization, we can provide inspiration for the further development of LMMs in terms of interpretability and generalization.