QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds

TL;DR

QuadrupedGPT is a versatile quadruped robotic agent that uses multimodal understanding and advanced reasoning to autonomously navigate, adapt, and execute complex, long-term tasks in open-ended environments.

Contribution

This work introduces QuadrupedGPT, a multimodal large model enabling quadruped robots to interpret commands, plan, and adapt for diverse tasks, advancing autonomous agility and cognition.

Findings

Proficient in handling diverse tasks and instructions

Effective multimodal observation utilization for decision-making

Demonstrates agility comparable to pets in complex scenarios

Abstract

As robotic agents increasingly assist humans in reality, quadruped robots offer unique opportunities for interaction in complex scenarios due to their agile movement. However, building agents that can autonomously navigate, adapt, and respond to versatile goals remains a significant challenge. In this work, we introduce QuadrupedGPT designed to follow diverse commands with agility comparable to that of a pet. The primary challenges addressed include: i) effectively utilizing multimodal observations for informed decision-making; ii) achieving agile control by integrating locomotion and navigation; iii) developing advanced cognition to execute long-term objectives. Our QuadrupedGPT interprets human commands and environmental contexts using a large multimodal model. Leveraging its extensive knowledge base, the agent autonomously assigns parameters for adaptive locomotion policies and devises safe yet efficient paths toward its goals. Additionally, it employs high-level reasoning to decompose long-term goals into a sequence of executable subgoals. Through comprehensive experiments, our agent shows proficiency in handling diverse tasks and intricate instructions, representing a significant step toward the development of versatile quadruped agents for open-ended environments.