FLYINGTRUST: A Benchmark for Quadrotor Navigation Across Scenarios and Vehicles

TL;DR

FLYINGTRUST is a comprehensive benchmarking framework for quadrotor navigation that evaluates how different vehicle capabilities and scene geometries impact algorithm robustness, aiding systematic comparison and development.

Contribution

The paper introduces FLYINGTRUST, a high-fidelity, configurable benchmark with standardized evaluation protocols for assessing quadrotor navigation across diverse scenarios and platforms.

Findings

Navigation success depends on platform capability and scene geometry.

Different algorithms show distinct failure modes and preferences.

Systematic patterns reveal the importance of considering both platform and scenario in evaluation.

Abstract

Visual navigation algorithms for quadrotors often exhibit a large variation in performance when transferred across different vehicle platforms and scene geometries, which increases the cost and risk of field deployment. To support systematic early-stage evaluation, we introduce FLYINGTRUST, a high-fidelity, configurable benchmarking framework that measures how platform kinodynamics and scenario structure jointly affect navigation robustness. FLYINGTRUST models vehicle capability with two compact, physically interpretable indicators: maximum thrust-to-weight ratio and axis-wise maximum angular acceleration. The benchmark pairs a diverse scenario library with a heterogeneous set of real and virtual platforms and prescribes a standardized evaluation protocol together with a composite scoring method that balances scenario importance, platform importance and performance stability. We use FLYINGTRUST to compare representative optimization-based and learning-based navigation approaches under identical conditions, performing repeated trials per platform-scenario combination and reporting uncertainty-aware metrics. The results reveal systematic patterns: navigation success depends predictably on platform capability and scene geometry, and different algorithms exhibit distinct preferences and failure modes across the evaluated conditions. These observations highlight the practical necessity of incorporating both platform capability and scenario structure into algorithm design, evaluation, and selection, and they motivate future work on methods that remain robust across diverse platforms and scenarios.