Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

TL;DR

This paper reviews the integration of large language models, especially multimodal ones like GPT-4V, into robotics, highlighting opportunities, challenges, and proposing a framework to improve embodied task planning with visual perception.

Contribution

It provides a comprehensive survey of LLMs in robotics and introduces a novel framework using multimodal GPT-4V to enhance embodied task planning.

Findings

GPT-4V improves robot performance in embodied tasks

Multimodal LLMs effectively integrate language and visual perception

Survey highlights key challenges and future directions in LLM-robot integration

Abstract

Large language models (LLMs) have undergone significant expansion and have been increasingly integrated across various domains. Notably, in the realm of robot task planning, LLMs harness their advanced reasoning and language comprehension capabilities to formulate precise and efficient action plans based on natural language instructions. However, for embodied tasks, where robots interact with complex environments, text-only LLMs often face challenges due to a lack of compatibility with robotic visual perception. This study provides a comprehensive overview of the emerging integration of LLMs and multimodal LLMs into various robotic tasks. Additionally, we propose a framework that utilizes multimodal GPT-4V to enhance embodied task planning through the combination of natural language instructions and robot visual perceptions. Our results, based on diverse datasets, indicate that GPT-4V effectively enhances robot performance in embodied tasks. This extensive survey and evaluation of LLMs and multimodal LLMs across a variety of robotic tasks enriches the understanding of LLM-centric embodied intelligence and provides forward-looking insights toward bridging the gap in Human-Robot-Environment interaction.