Grounding Large Language Models In Embodied Environment With Imperfect World Models

TL;DR

This paper introduces GLIMO, a method that grounds large language models in physical environments using imperfect world models and an automated data generation process, significantly improving their physical reasoning and robotics capabilities.

Contribution

The paper presents a novel framework that combines proxy world models with an LLM agent-based data generator to enhance physical reasoning in LLMs, outperforming existing models on multiple benchmarks.

Findings

Performance improved by up to 2.04 times on benchmark tasks.

LLaMA-3 models surpass larger models like GPT-4 in specific tasks.

Automated data generation enhances physical reasoning in LLMs.

Abstract

Despite a widespread success in various applications, large language models (LLMs) often stumble when tackling basic physical reasoning or executing robotics tasks, due to a lack of direct experience with the physical nuances of the real world. To address these issues, we propose a Grounding Large language model with Imperfect world MOdel (GLIMO), which utilizes proxy world models such as simulators to collect and synthesize trining data. GLIMO incorporates an LLM agent-based data generator to automatically create high-quality and diverse instruction datasets. The generator includes an iterative self-refining module for temporally consistent experience sampling, a diverse set of question-answering instruction seeds, and a retrieval-augmented generation module for reflecting on prior experiences. Comprehensive experiments show that our approach improve the performance of strong open-source LLMs like LLaMA-3 with a performance boost of 2.04 $\times$, 1.54 $\times$, and 1.82 $\times$ across three different benchmarks, respectively. The performance is able to compete with or surpass their larger counterparts such as GPT-4.