Analysis and Detection of Differences in Spoken User Behaviors between Autonomous and Wizard-of-Oz Systems

TL;DR

This paper investigates behavioral differences in spoken interactions with autonomous versus operator-controlled robots, analyzing speech patterns and developing models to distinguish between the two conditions.

Contribution

It provides a comprehensive analysis of user speech behaviors and introduces predictive models that outperform baselines in identifying system autonomy.

Findings

Significant differences in speech metrics between conditions

Predictive models achieved higher accuracy and F1 scores

Behavioral markers can reliably distinguish autonomous from operator-controlled interactions

Abstract

This study examined users' behavioral differences in a large corpus of Japanese human-robot interactions, comparing interactions between a tele-operated robot and an autonomous dialogue system. We analyzed user spoken behaviors in both attentive listening and job interview dialogue scenarios. Results revealed significant differences in metrics such as speech length, speaking rate, fillers, backchannels, disfluencies, and laughter between operator-controlled and autonomous conditions. Furthermore, we developed predictive models to distinguish between operator and autonomous system conditions. Our models demonstrated higher accuracy and precision compared to the baseline model, with several models also achieving a higher F1 score than the baseline.