A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks

TL;DR

This paper introduces a multimodal corpus capturing expert gaze and behavior during phonetic segmentation tasks to improve automatic speech segmentation accuracy by modeling human manual segmentation processes.

Contribution

It provides a new multimodal dataset of expert behavior during phonetic segmentation, enabling better modeling of manual segmentation for automatic systems.

Findings

Corpus captures visual and auditory cues used by experts

Data highlights key features of manual segmentation process

Potential to improve automatic segmentation accuracy

Abstract

Phonetic segmentation is the process of splitting speech into distinct phonetic units. Human experts routinely perform this task manually by analyzing auditory and visual cues using analysis software, which is an extremely time-consuming process. Methods exist for automatic segmentation, but these are not always accurate enough. In order to improve automatic segmentation, we need to model it as close to the manual segmentation as possible. This corpus is an effort to capture the human segmentation behavior by recording experts performing a segmentation task. We believe that this data will enable us to highlight the important aspects of manual segmentation, which can be used in automatic segmentation to improve its accuracy.