MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Models

TL;DR

This paper introduces MPCC, a new benchmark for evaluating multimodal large language models' ability to handle complex, real-world planning tasks with multiple constraints across modalities, revealing significant challenges and limitations.

Contribution

The paper presents MPCC, the first benchmark to systematically assess MLLMs' multimodal planning with complex constraints, including tasks, graded difficulty levels, and evaluation of current model performance.

Findings

Models perform poorly on complex constrained planning tasks.

Traditional prompting strategies fail in multi-constraint scenarios.

Models are highly sensitive to the complexity of constraints.

Abstract

Multimodal planning capabilities refer to the ability to predict, reason, and design steps for task execution with multimodal context, which is essential for complex reasoning and decision-making across multiple steps. However, current benchmarks face two key challenges: (1) they cannot directly assess multimodal real-world planning capabilities, and (2) they lack constraints or implicit constraints across modalities. To address these issues, we introduce Multimodal Planning with Complex Constraints (MPCC), the first benchmark to systematically evaluate MLLMs' ability to handle multimodal constraints in planning. To address the first challenge, MPCC focuses on three real-world tasks: Flight Planning, Calendar Planning, and Meeting Planning. To solve the second challenge, we introduce complex constraints (e.g. budget, temporal, and spatial) in these tasks, with graded difficulty levels (EASY, MEDIUM, HARD) to separate constraint complexity from search space expansion. Experiments on 13 advanced MLLMs reveal significant challenges: closed-source models achieve only 21.3% feasible plans, while open-source models average below 11%. Additionally, we observe that MLLMs are highly sensitive to constraint complexity and that traditional multimodal prompting strategies fail in multi-constraint scenarios. Our work formalizes multimodal constraints in planning, provides a rigorous evaluation framework, and highlights the need for advancements in constraint-aware reasoning for real-world MLLM applications.