Open Grounded Planning: Challenges and Benchmark Construction

TL;DR

This paper introduces open grounded planning, a new task for generating executable plans based on variable action sets, and constructs a benchmark to evaluate LLMs and planning methods across diverse domains.

Contribution

It defines the open grounded planning task, creates a comprehensive benchmark dataset, and evaluates current LLMs, highlighting existing challenges and future research directions.

Findings

Current LLMs struggle with open grounded planning tasks.

Existing methods are insufficient for generating executable plans in open domains.

The benchmark reveals significant gaps in LLM capabilities for grounded planning.

Abstract

The emergence of large language models (LLMs) has increasingly drawn attention to the use of LLMs for human-like planning. Existing work on LLM-based planning either focuses on leveraging the inherent language generation capabilities of LLMs to produce free-style plans, or employs reinforcement learning approaches to learn decision-making for a limited set of actions within restricted environments. However, both approaches exhibit significant discrepancies from the open and executable requirements in real-world planning. In this paper, we propose a new planning task--open grounded planning. The primary objective of open grounded planning is to ask the model to generate an executable plan based on a variable action set, thereby ensuring the executability of the produced plan. To this end, we establishes a benchmark for open grounded planning spanning a wide range of domains. Then we test current state-of-the-art LLMs along with five planning approaches, revealing that existing LLMs and methods still struggle to address the challenges posed by grounded planning in open domains. The outcomes of this paper define and establish a foundational dataset for open grounded planning, and shed light on the potential challenges and future directions of LLM-based planning.