The Design of Informative Take-Over Requests for Semi-Autonomous Cyber-Physical Systems: Combining Spoken Language and Visual Icons in a Drone-Controller Setting

TL;DR

This study designs and evaluates a multimodal take-over request system for semi-autonomous drones, combining speech and visual cues to improve human understanding and response accuracy during control handovers.

Contribution

It introduces a novel bi-modal TOR design that integrates spoken language and visual highlights, tested in a drone control scenario, with insights on timing and message length effects.

Findings

Bi-modal TOR improved accuracy and situation recognition.

Shorter spoken fragments did not enhance performance.

Synchronous visual highlighting did not improve accuracy and increased response times.

Abstract

The question of how cyber-physical systems should interact with human partners that can take over control or exert oversight is becoming more pressing, as these systems are deployed for an ever larger range of tasks. Drawing on the literatures on handing over control during semi-autonomous driving and human-robot interaction, we propose a design of a take-over request that combines an abstract pre-alert with an informative TOR: Relevant sensor information is highlighted on the controller's display, while a spoken message verbalizes the reason for the TOR. We conduct our study in the context of a semi-autonomous drone control scenario as our testbed. The goal of our online study is to assess in more detail what form a language-based TOR should take. Specifically, we compare a full sentence condition to shorter fragments, and test whether the visual highlighting should be done synchronously or asynchronously with the speech. Participants showed a higher accuracy in choosing the correct solution with our bi-modal TOR and felt that they were better able to recognize the critical situation. Using only fragments in the spoken message rather than full sentences did not lead to improved accuracy or faster reactions. Also, synchronizing the visual highlighting with the spoken message did not result in better accuracy and response times were even increased in this condition.