A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding

TL;DR

A-MAR introduces an agent-based framework for multimodal art retrieval that explicitly plans reasoning steps, improving interpretability and evidence grounding in artwork understanding.

Contribution

It presents a novel structured reasoning plan approach for multimodal art retrieval, enhancing explainability and multi-step reasoning capabilities.

Findings

A-MAR outperforms static retrieval and baseline models in explanation quality.

It demonstrates improved evidence grounding and reasoning on ArtCoT-QA.

Code and data are publicly available at the provided GitHub link.

Abstract

Understanding artworks requires multi-step reasoning over visual content and cultural, historical, and stylistic context. While recent multimodal large language models show promise in artwork explanation, they rely on implicit reasoning and internalized knowl- edge, limiting interpretability and explicit evidence grounding. We propose A-MAR, an Agent-based Multimodal Art Retrieval framework that explicitly conditions retrieval on structured reasoning plans. Given an artwork and a user query, A-MAR first decomposes the task into a structured reasoning plan that specifies the goals and evidence requirements for each step. Retrieval is then conditionedon this plan, enabling targeted evidence selection and supporting step-wise, grounded explanations. To evaluate agent-based multi- modal reasoning within the art domain, we introduce ArtCoT-QA. This diagnostic benchmark features multi-step reasoning chains for diverse art-related queries, enabling a granular analysis that extends beyond simple final answer accuracy. Experiments on SemArt and Artpedia show that A-MAR consistently outperforms static, non planned retrieval and strong MLLM baselines in final explanation quality, while evaluations on ArtCoT-QA further demonstrate its advantages in evidence grounding and multi-step reasoning ability. These results highlight the importance of reasoning-conditioned retrieval for knowledge-intensive multimodal understanding and position A-MAR as a step toward interpretable, goal-driven AI systems, with particular relevance to cultural industries. The code and data are available at: https://github.com/ShuaiWang97/A-MAR.