M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark

TL;DR

M^3-Bench is a comprehensive benchmark designed to evaluate multimodal, multi-hop, multi-threaded tool use in large language models, emphasizing realistic workflows, visual grounding, and reasoning across tools.

Contribution

It introduces a novel similarity-driven alignment method and standardized evaluation pipeline for assessing complex multimodal tool-using workflows in LLMs.

Findings

State-of-the-art models show gaps in argument fidelity.

Workflow structure consistency is often lacking.

Benchmark reveals areas for improvement in multimodal reasoning.

Abstract

We present M^3-Bench, the first benchmark for evaluating multimodal tool use under the Model Context Protocol. The benchmark targets realistic, multi-hop and multi-threaded workflows that require visual grounding and textual reasoning, cross-tool dependencies, and persistence of intermediate resources across steps. We introduce a similarity-driven alignment that serializes each tool call, embeds signatures with a sentence encoder, and performs similarity-bucketed Hungarian matching to obtain auditable one-to-one correspondences. On top of this alignment, we report interpretable metrics that decouple semantic fidelity from workflow consistency. The benchmark spans 28 servers with 231 tools, and provides standardized trajectories curated through an Executor & Judge pipeline with human verification; an auxiliary four large language models (LLMs) judge ensemble reports end-task Task Completion and information grounding. Evaluations of representative state-of-the-art Multimodal LLMs (MLLMs) reveal persistent gaps in multimodal MCP tool use, particularly in argument fidelity and structure consistency, underscoring the need for methods that jointly reason over images, text, and tool graphs. Our Benchmark's anonymous repository is at https://github.com/EtaYang10th/Open-M3-Bench