Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics

TL;DR

This paper introduces a differentiable physics-based deep learning approach for agile, high-speed vision-based drone navigation in complex environments, achieving high success rates and autonomous coordination without extensive hardware or communication.

Contribution

It combines simple physics simulation with neural network training for robust, real-time drone navigation, enabling zero-shot sim-to-real transfer and multi-agent coordination.

Findings

90% success rate in complex environments

Operates at speeds up to 20 m/s in real-world forests

Achieves autonomous multi-agent coordination without communication

Abstract

Swarm navigation in cluttered environments is a grand challenge in robotics. This work combines deep learning with first-principle physics through differentiable simulation to enable autonomous navigation of multiple aerial robots through complex environments at high speed. Our approach optimizes a neural network control policy directly by backpropagating loss gradients through the robot simulation using a simple point-mass physics model and a depth rendering engine. Despite this simplicity, our method excels in challenging tasks for both multi-agent and single-agent applications with zero-shot sim-to-real transfer. In multi-agent scenarios, our system demonstrates self-organized behavior, enabling autonomous coordination without communication or centralized planning - an achievement not seen in existing traditional or learning-based methods. In single-agent scenarios, our system achieves a 90% success rate in navigating through complex environments, significantly surpassing the 60% success rate of the previous state-of-the-art approach. Our system can operate without state estimation and adapt to dynamic obstacles. In real-world forest environments, it navigates at speeds up to 20 m/s, doubling the speed of previous imitation learning-based solutions. Notably, all these capabilities are deployed on a budget-friendly $21 computer, costing less than 5% of a GPU-equipped board used in existing systems. Video demonstrations are available at https://youtu.be/LKg9hJqc2cc.