RoboGPT: an intelligent agent of making embodied long-term decisions for daily instruction tasks

TL;DR

RoboGPT is a novel robotic agent that combines large language models with re-planning and specialized skills to improve long-term decision-making and task execution in daily instruction tasks, outperforming existing methods.

Contribution

The paper introduces RoboGPT, integrating LLM-based planning with a re-plan module and specialized RoboSkills, supported by a new robotic dataset, to enhance feasibility and adaptability in robotic task planning.

Findings

Outperforms SOTA on ALFRED daily tasks

Exceeds SOTA LLM planners in task rationality

Generalizes well to unseen tasks

Abstract

Robotic agents must master common sense and long-term sequential decisions to solve daily tasks through natural language instruction. The developments in Large Language Models (LLMs) in natural language processing have inspired efforts to use LLMs in complex robot planning. Despite LLMs' great generalization and comprehension of instruction tasks, LLMs-generated task plans sometimes lack feasibility and correctness. To address the problem, we propose a RoboGPT agent\footnote{our code and dataset will be released soon} for making embodied long-term decisions for daily tasks, with two modules: 1) LLMs-based planning with re-plan to break the task into multiple sub-goals; 2) RoboSkill individually designed for sub-goals to learn better navigation and manipulation skills. The LLMs-based planning is enhanced with a new robotic dataset and re-plan, called RoboGPT. The new robotic dataset of 67k daily instruction tasks is gathered for fine-tuning the Llama model and obtaining RoboGPT. RoboGPT planner with strong generalization can plan hundreds of daily instruction tasks. Additionally, a low-computational Re-Plan module is designed to allow plans to flexibly adapt to the environment, thereby addressing the nomenclature diversity challenge. The proposed RoboGPT agent outperforms SOTA methods on the ALFRED daily tasks. Moreover, RoboGPT planner exceeds SOTA LLM-based planners like ChatGPT in task-planning rationality for hundreds of unseen daily tasks, and even other domain tasks, while keeping the large model's original broad application and generality.