FlightBench: Benchmarking Learning-based Methods for Ego-vision-based Quadrotors Navigation
Shu-Ang Yu, Chao Yu, Feng Gao, Yi Wu, Yu Wang
TL;DR
FlightBench provides a comprehensive benchmark for evaluating learning-based ego-vision navigation methods in quadrotors, comparing them with optimization-based approaches across various scenarios and difficulty levels, highlighting their strengths and limitations.
Contribution
This paper introduces FlightBench, the first benchmark for comparing learning-based and optimization-based ego-vision navigation methods in quadrotors, with new scenario difficulty criteria.
Findings
Learning-based methods excel in high-speed flight and inference speed.
They struggle with challenging scenarios like sharp corners and occlusions.
Difficulty criteria correlate well with flight performance.
Abstract
Ego-vision-based navigation in cluttered environments is crucial for mobile systems, particularly agile quadrotors. While learning-based methods have shown promise recently, head-to-head comparisons with cutting-edge optimization-based approaches are scarce, leaving open the question of where and to what extent they truly excel. In this paper, we introduce FlightBench, the first comprehensive benchmark that implements various learning-based methods for ego-vision-based navigation and evaluates them against mainstream optimization-based baselines using a broad set of performance metrics. More importantly, we develop a suite of criteria to assess scenario difficulty and design test cases that span different levels of difficulty based on these criteria. Our results show that while learning-based methods excel in high-speed flight and faster inference, they struggle with challenging…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
### Novel benchmark - unified open-source benchmark that enables direct comparison between learning-based and optimization-based methods for UAV navigation - 3 new quantitative metrics for measuring scenario difficulty - evaluation across multiple scenarios with varying difficulty levels ### Paper - well written and illustrated, only minor errors falling through
### Scope / contribution - engineering work, the paper appears to be a mix of experiments with existing methods; 3 task difficulty metrics are not enough for ICLR - out of scope, more suitable for a robotics conference such as ICRA ### Benchmark issues - limited number of learning-based methods evaluated - limited to simulated environments
1. The paper is generally well-written, and well motivates the need for a 3D scene based benchmark for ego-vision agile navigation, which is currently unaddressed by other benchmarks. 2. The 3 proposed task-difficulty metrics, can comprehensively capture and quantify the challenge of a particular scene with obstacles. In addition, the proposed scenarios are diverse in terms of the task difficulty metrics and capture the different operating conditions often faced. 3. A number of SOTA baselines
1. While the benchmark introduces a variety of scenes, it is still quite limited - as a dataset benchmark, I would have expected more scenarios. 2. Furthermore, the authors do not discuss and analyze the number of scenarios compared to previous baselines. It would be good to have this in the paper. The authors should also discuss the diversity of previous baselines in terms of their proposed task difficulty metrics. 3. Gazebo and ROS-Noetic is used as the simulation platform, which often does
- The motivation to bring together different methods (learnable and not) for vision-based UAV navigation and compare them head-to-head, addresses an important lack of standardization in the field, that can drive further progress. - The selected testing environments are quite representative of a wide range of different flying scenarios. - The evaluation section offers valuable insights on the pros and cons of both categories of UAV navigation solutions, which is very informative and can drive fu
- The majority of metrics and methods considered in the proposed baseline have been proposed above, or widely used in the community. This may limit the novelty of the proposed approach. However, in my opinion its contribution remains unaffected, as it still provides a comprehensive suite for uniformly evaluating different navigation approaches. - The proposed benchmark remains rather targeted on static environments and sensing modalities. Driven by the easy extendibility of the proposed framewo
1. This benchmark has capability of 3D scenarios, classical methods and learning methods for planning while allowing sensory inputs in form of vision. 2. Three different scenarios with eight difficulty level has been presented. 3. Performance on different computing platforms have been shown.
1. The overall paper's core contribution is unclear, i.e. whether it is on the technological side, creating virtual scenarios, etc. 2. It seems this paper has proposed three simulation scenarios with difficulty levels based on different evaluation metrics. However, I don't see any novelty or crucial research contribution from the ICLR perspective. In other words, there are no theoretical or experimental contributions. The paper appears as a system paper where multiple things are simply combined
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRobotic Path Planning Algorithms · Robotics and Sensor-Based Localization · Air Traffic Management and Optimization