Influence of visual cues on head and eye movements during listening tasks in multi-talker audiovisual environments with animated characters

TL;DR

This study investigates how visual cues from animated characters in virtual environments affect head and eye movements, task performance, and perception during listening tasks with young normal hearing individuals.

Contribution

It provides new insights into how animated visual cues influence movement and perception, supporting their use in assessing hearing aid benefits.

Findings

Visual cues significantly affect head and eye movements.

Lip-syncing animations elicit similar movement patterns to real videos.

Visual cues influence perceived speech intelligibility and listening effort.

Abstract

Recent studies of hearing aid benefits indicate that head movement behavior influences performance. To systematically assess these effects, movement behavior must be measured in realistic communication conditions. For this, the use of virtual audiovisual environments with animated characters as visual stimuli has been proposed. It is unclear, however, how these animations influence the head- and eye-movement behavior of subjects. Here, two listening tasks were carried out with a group of 14 young normal hearing subjects to investigate the influence of visual cues on head- and eye-movement behavior; on combined localization and speech intelligibility task performance; as well as on perceived speech intelligibility, perceived listening effort and the general impression of the audiovisual environments. Animated characters with different lip-syncing and gaze patterns were compared to an audio-only condition and to a video of real persons. Results show that movement behavior, task performance, and perception were all influenced by visual cues. The movement behavior of young normal hearing listeners in animation conditions with lip-syncing was similar to that in the video condition. These results in young normal hearing listeners are a first step towards using the animated characters to assess the influence of head movement behavior on hearing aid performance.