RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Models

TL;DR

RobotDesignGPT automates robot design synthesis by leveraging vision-language models, enabling the creation of visually appealing and kinematically valid robots from simple prompts and reference images, reducing manual effort.

Contribution

The paper introduces a novel framework that uses large pre-trained vision-language models for automated robot design, incorporating a visual feedback mechanism to enhance quality.

Findings

Designs inspired by nature, including legged and flying robots.

Improved design quality with visual feedback approach.

Validated through ablation and user studies.

Abstract

Robot design is a nontrivial process that involves careful consideration of multiple criteria, including user specifications, kinematic structures, and visual appearance. Therefore, the design process often relies heavily on domain expertise and significant human effort. The majority of current methods are rule-based, requiring the specification of a grammar or a set of primitive components and modules that can be composed to create a design. We propose a novel automated robot design framework, RobotDesignGPT, that leverages the general knowledge and reasoning capabilities of large pre-trained vision-language models to automate the robot design synthesis process. Our framework synthesizes an initial robot design from a simple user prompt and a reference image. Our novel visual feedback approach allows us to greatly improve the design quality and reduce unnecessary manual feedback. We demonstrate that our framework can design visually appealing and kinematically valid robots inspired by nature, ranging from legged animals to flying creatures. We justify the proposed framework by conducting an ablation study and a user study.