A virtual reality-based method for examining audiovisual prosody perception

TL;DR

This paper introduces a virtual reality method to study audiovisual prosody perception, highlighting how virtual human animations can simulate real speaker cues and facilitate multimodal communication research.

Contribution

It presents a novel VR-based approach for examining audiovisual prosody, enabling controlled studies of multimodal effects in verbal communication.

Findings

Virtual human animations mimic real speaker cues

VR method facilitates multimodal prosody research

Potential applications for cochlear implant studies

Abstract

Prosody plays a vital role in verbal communication. Acoustic cues of prosody have been examined extensively. However, prosodic characteristics are not only perceived auditorily, but also visually based on head and facial movements. The purpose of this report is to present a method for examining audiovisual prosody using virtual reality. We show that animations based on a virtual human provide motion cues similar to those obtained from video recordings of a real talker. The use of virtual reality opens up new avenues for examining multimodal effects of verbal communication. We discuss the method in the framework of examining prosody perception in cochlear implant listeners.