Agile Flight Emerges from Multi-Agent Competitive Racing

TL;DR

This paper demonstrates that multi-agent competitive reinforcement learning enables the emergence of agile flight and strategic behaviors in agents, outperforming traditional single-agent training especially in complex environments, with better transfer to real-world scenarios.

Contribution

It shows that sparse, high-level rewards in multi-agent competition can produce advanced physical control behaviors and improve sim-to-real transfer, surpassing traditional reward-based training methods.

Findings

Multi-agent competition leads to emergent agile flight behaviors.

Policies trained with multi-agent competition transfer more reliably to real-world robots.

Multi-agent policies generalize better to unseen opponents.

Abstract

Through multi-agent competition and the sparse high-level objective of winning a race, we find that both agile flight (e.g., high-speed motion pushing the platform to its physical limits) and strategy (e.g., overtaking or blocking) emerge from agents trained with reinforcement learning. We provide evidence in both simulation and the real world that this approach outperforms the common paradigm of training agents in isolation with rewards that prescribe behavior, e.g., progress on the raceline, in particular when the complexity of the environment increases, e.g., in the presence of obstacles. Moreover, we find that multi-agent competition yields policies that transfer more reliably to the real world than policies trained with a single-agent progress-based reward, despite the two methods using the same simulation environment, randomization strategy, and hardware. In addition to improved sim-to-real transfer, the multi-agent policies also exhibit some degree of generalization to opponents unseen at training time. Overall, our work, following in the tradition of multi-agent competitive game-play in digital domains, shows that sparse task-level rewards are sufficient for training agents capable of advanced low-level control in the physical world. Code: https://github.com/Jirl-upenn/AgileFlight_MultiAgent