DECISIVE Benchmarking Data Report: sUAS Performance Results from Phase I

TL;DR

This report presents comprehensive benchmarking results of eight small unmanned aerial systems (sUAS) in subterranean environments, evaluating their performance across 20 test methods to identify top performers and provide a detailed performance overview.

Contribution

It introduces a standardized benchmarking framework for sUAS in subterranean settings and provides the first extensive performance comparison across multiple platforms.

Findings

Identified top-performing sUAS in various test categories.

Provided detailed performance metrics for 8 different sUAS platforms.

Established criteria for best-in-class performance in subterranean environments.

Abstract

This report reviews all results derived from performance benchmarking conducted during Phase I of the Development and Execution of Comprehensive and Integrated Subterranean Intelligent Vehicle Evaluations (DECISIVE) project by the University of Massachusetts Lowell, using the test methods specified in the DECISIVE Test Methods Handbook v1.1 for evaluating small unmanned aerial systems (sUAS) performance in subterranean and constrained indoor environments, spanning communications, field readiness, interface, obstacle avoidance, navigation, mapping, autonomy, trust, and situation awareness. Using those 20 test methods, over 230 tests were conducted across 8 sUAS platforms: Cleo Robotics Dronut X1P (P = prototype), FLIR Black Hornet PRS, Flyability Elios 2 GOV, Lumenier Nighthawk V3, Parrot ANAFI USA GOV, Skydio X2D, Teal Golden Eagle, and Vantage Robotics Vesper. Best in class criteria is specified for each applicable test method and the sUAS that match this criteria are named for each test method, including a high-level executive summary of their performance.