Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design

TL;DR

Swarm-GPT combines large language models with safe motion planning to automatically generate and execute synchronized drone choreographies based on natural language instructions, ensuring safety and realism.

Contribution

This work introduces Swarm-GPT, a novel system integrating LLMs with collision-free swarm motion planning for drone choreography design.

Findings

Achieved a mean sim-to-real RMSE of 28.7 mm in trajectory transfer.

Successfully demonstrated at three live events.

Enables natural language-based drone choreography generation.

Abstract

This paper presents Swarm-GPT, a system that integrates large language models (LLMs) with safe swarm motion planning - offering an automated and novel approach to deployable drone swarm choreography. Swarm-GPT enables users to automatically generate synchronized drone performances through natural language instructions. With an emphasis on safety and creativity, Swarm-GPT addresses a critical gap in the field of drone choreography by integrating the creative power of generative models with the effectiveness and safety of model-based planning algorithms. This goal is achieved by prompting the LLM to generate a unique set of waypoints based on extracted audio data. A trajectory planner processes these waypoints to guarantee collision-free and feasible motion. Results can be viewed in simulation prior to execution and modified through dynamic re-prompting. Sim-to-real transfer experiments demonstrate Swarm-GPT's ability to accurately replicate simulated drone trajectories, with a mean sim-to-real root mean square error (RMSE) of 28.7 mm. To date, Swarm-GPT has been successfully showcased at three live events, exemplifying safe real-world deployment of pre-trained models.