Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios

TL;DR

This paper introduces DailyClue, a benchmark dataset designed to evaluate multimodal large language models' ability to perform visual clue-driven reasoning in daily scenarios, emphasizing active exploration and reasoning beyond surface perception.

Contribution

The paper presents a new benchmark with a comprehensive dataset across daily domains, focusing on reasoning that requires identifying and leveraging visual clues, filling a gap in existing evaluations.

Findings

Current MLLMs struggle with visual clue-driven reasoning.

Accurate visual clue identification is crucial for robust reasoning.

Benchmark reveals significant challenges for existing models.

Abstract

Daily scenarios are characterized by visual richness, requiring Multimodal Large Language Models (MLLMs) to filter noise and identify decisive visual clues for accurate reasoning. Yet, current benchmarks predominantly aim at evaluating MLLMs' pre-existing knowledge or perceptual understanding, often neglecting the critical capability of reasoning. To bridge this gap, we introduce DailyClue, a benchmark designed for visual clue-driven reasoning in daily scenarios. Our construction is guided by two core principles: (1) strict grounding in authentic daily activities, and (2) challenging query design that necessitates more than surface-level perception. Instead of simple recognition, our questions compel MLLMs to actively explore suitable visual clues and leverage them for subsequent reasoning. To this end, we curate a comprehensive dataset spanning four major daily domains and 16 distinct subtasks. Comprehensive evaluation across MLLMs and agentic models underscores the formidable challenge posed by our benchmark. Our analysis reveals several critical insights, emphasizing that the accurate identification of visual clues is essential for robust reasoning.