Did somebody say "Gest-IT"? A pilot exploration of multimodal data management

TL;DR

This paper explores the creation and analysis of a multimodal corpus with orthographic, prosodic, and gestural annotations to study gesture patterns in conversations involving sighted and visually impaired individuals.

Contribution

It introduces the Gest-IT resource, a multimodal corpus with a three-layer annotation scheme, and discusses transcription methods and future development steps.

Findings

Gesture patterns vary between sighted and visually impaired speakers

The Gest-IT corpus enables detailed multimodal analysis

A unified CoNLL-U format facilitates data sharing and analysis

Abstract

The paper presents a pilot exploration of the construction, management and analysis of a multimodal corpus. Through a three-layer annotation that provides orthographic, prosodic, and gestural transcriptions, the Gest-IT resource allows to investigate the variation of gesture-making patterns in conversations between sighted people and people with visual impairment. After discussing the transcription methods and technical procedures employed in our study, we propose a unified CoNLL-U corpus and indicate our future steps