m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks

TL;DR

This paper introduces m&m's, a comprehensive benchmark with over 4,000 multi-step multi-modal tasks and tools, to evaluate and improve LLM-based planning strategies for complex real-world problems.

Contribution

It provides a large, annotated dataset and systematic evaluation framework for assessing LLMs as multi-step multi-modal task planners, addressing a key gap in the field.

Findings

Multi-step planning generally outperforms single-shot planning.

Structured data formats like JSON improve plan clarity and execution.

Feedback mechanisms enhance planning accuracy.

Abstract

Real-world multi-modal problems are rarely solved by a single machine learning model, and often require multi-step computational plans that involve stitching several models. Tool-augmented LLMs hold tremendous promise for automating the generation of such computational plans. However, the lack of standardized benchmarks for evaluating LLMs as planners for multi-step multi-modal tasks has prevented a systematic study of planner design decisions. Should LLMs generate a full plan in a single shot or step-by-step? Should they invoke tools directly with Python code or through structured data formats like JSON? Does feedback improve planning? To answer these questions and more, we introduce m&m's: a benchmark containing 4K+ multi-step multi-modal tasks involving 33 tools that include multi-modal models, (free) public APIs, and image processing modules. For each of these task queries, we provide automatically generated plans using this realistic toolset. We further provide a high-quality subset of 1,565 task plans that are human-verified and correctly executable. With m&m's, we evaluate 10 popular LLMs with 2 planning strategies (multi-step vs. step-by-step planning), 2 plan formats (JSON vs. code), and 3 types of feedback (parsing/verification/execution). Finally, we summarize takeaways from our extensive experiments. Our dataset and code are available on HuggingFace (https://huggingface.co/datasets/zixianma/mnms) and Github (https://github.com/RAIVNLab/mnms).