InsightVision: A Comprehensive, Multi-Level Chinese-based Benchmark for Evaluating Implicit Visual Semantics in Large Vision Language Models

TL;DR

This paper introduces InsightVision, a new Chinese-based benchmark for evaluating the understanding of implicit visual semantics in large vision language models, revealing current models' limitations compared to humans.

Contribution

It presents a comprehensive, multi-level benchmark for implicit visual semantics in Chinese, along with a semi-automatic dataset construction method and evaluation of 15 LVLMs and GPT-4o.

Findings

Models lag nearly 14% behind human performance in implicit understanding.

Current LVLMs struggle with nuanced visual semantics.

The benchmark covers four levels of implicit meaning comprehension.

Abstract

In the evolving landscape of multimodal language models, understanding the nuanced meanings conveyed through visual cues - such as satire, insult, or critique - remains a significant challenge. Existing evaluation benchmarks primarily focus on direct tasks like image captioning or are limited to a narrow set of categories, such as humor or satire, for deep semantic understanding. To address this gap, we introduce, for the first time, a comprehensive, multi-level Chinese-based benchmark designed specifically for evaluating the understanding of implicit meanings in images. This benchmark is systematically categorized into four subtasks: surface-level content understanding, symbolic meaning interpretation, background knowledge comprehension, and implicit meaning comprehension. We propose an innovative semi-automatic method for constructing datasets, adhering to established construction protocols. Using this benchmark, we evaluate 15 open-source large vision language models (LVLMs) and GPT-4o, revealing that even the best-performing model lags behind human performance by nearly 14% in understanding implicit meaning. Our findings underscore the intrinsic challenges current LVLMs face in grasping nuanced visual semantics, highlighting significant opportunities for future research and development in this domain. We will publicly release our InsightVision dataset, code upon acceptance of the paper.