LLM-BABYBENCH: Understanding and Evaluating Grounded Planning and Reasoning in LLMs

TL;DR

LLM-BabyBench is a new benchmark suite designed to evaluate large language models on grounded planning and reasoning tasks within a text-based environment, addressing key aspects of grounded intelligence.

Contribution

This paper introduces LLM-BabyBench, a comprehensive benchmark with datasets, evaluation metrics, and code for assessing LLMs' grounded reasoning in interactive environments, which was previously lacking.

Findings

Baseline results reveal significant challenges for LLMs in grounded reasoning tasks.

The benchmark provides a standardized framework for reproducible evaluation.

Datasets and tools are publicly available for the research community.

Abstract

Assessing the capacity of Large Language Models (LLMs) to plan and reason within the constraints of interactive environments is crucial for developing capable AI agents. We introduce $\textbf{LLM-BabyBench}$, a new benchmark suite designed specifically for this purpose. Built upon a textual adaptation of the procedurally generated BabyAI grid world, this suite evaluates LLMs on three fundamental aspects of grounded intelligence: (1) predicting the consequences of actions on the environment state ($\textbf{Predict}$ task), (2) generating sequences of low-level actions to achieve specified objectives ($\textbf{Plan}$ task), and (3) decomposing high-level instructions into coherent subgoal sequences ($\textbf{Decompose}$ task). We detail the methodology for generating the three corresponding datasets ($\texttt{LLM-BabyBench-Predict}$, $\texttt{-Plan}$, $\texttt{-Decompose}$) by extracting structured information from an expert agent operating within the text-based environment. Furthermore, we provide a standardized evaluation harness and metrics, including environment interaction for validating generated plans, to facilitate reproducible assessment of diverse LLMs. Initial baseline results highlight the challenges posed by these grounded reasoning tasks. The benchmark suite, datasets, data generation code, and evaluation code are made publicly available ($\href{https://github.com/choukrani/llm-babybench}{\text{GitHub}}$, $\href{https://huggingface.co/datasets/salem-mbzuai/LLM-BabyBench}{\text{HuggingFace}}$).