Towards Reproducible Evaluations for Flying Drone Controllers in Virtual Environments

TL;DR

This paper introduces VRFlightSim, a virtual environment for standardized evaluation of drone control interfaces, addressing the challenge of inconsistent physical environment testing in NUI research.

Contribution

It presents a new virtual platform for reproducible drone interface evaluation and demonstrates its utility through replication and user studies.

Findings

VRFlightSim enables consistent performance comparisons.

Replicated a state-of-the-art interface successfully.

User studies highlight the platform's potential for open-data interface design.

Abstract

Research attention on natural user interfaces (NUIs) for drone flights are rising. Nevertheless, NUIs are highly diversified, and primarily evaluated by different physical environments leading to hard-to-compare performance between such solutions. We propose a virtual environment, namely VRFlightSim, enabling comparative evaluations with enriched drone flight details to address this issue. We first replicated a state-of-the-art (SOTA) interface and designed two tasks (crossing and pointing) in our virtual environment. Then, two user studies with 13 participants demonstrate the necessity of VRFlightSim and further highlight the potential of open-data interface designs.