NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?

TL;DR

This paper introduces NEMO, a benchmark to evaluate multimodal large language models' ability to recognize attribute-modified objects, revealing significant performance gaps and insights into their limitations.

Contribution

The paper presents NEMO, a novel benchmark with 900 images and 2,700 questions, to systematically assess MLLMs' reasoning in recognizing attribute-modified objects.

Findings

MLLMs show notable performance gaps on NEMO.

Stronger vision encoders improve recognition, but MLLMs still lag behind standalone vision models.

Larger LLMs can weaken vision encoders during fine-tuning.

Abstract

Multimodal Large Language Models (MLLMs) have made notable advances in visual understanding, yet their abilities to recognize objects modified by specific attributes remain an open question. To address this, we explore MLLMs' reasoning capabilities in object recognition, ranging from commonsense to beyond-commonsense scenarios. We introduce a novel benchmark, NEMO, which comprises 900 images of origiNal fruits and their corresponding attributE-MOdified ones; along with a set of 2,700 questions including open-, multiple-choice-, unsolvable types. We assess 26 recent open-sourced and commercial models using our benchmark. The findings highlight pronounced performance gaps in recognizing objects in NEMO and reveal distinct answer preferences across different models. Although stronger vision encoders improve performance, MLLMs still lag behind standalone vision encoders. Interestingly, scaling up the model size does not consistently yield better outcomes, as deeper analysis reveals that larger LLMs can weaken vision encoders during fine-tuning. These insights shed light on critical limitations in current MLLMs and suggest potential pathways toward developing more versatile and resilient multimodal models.