Addressing the Blind Spots in Spoken Language Processing

TL;DR

This paper emphasizes the importance of non-verbal cues like gestures and facial expressions in human communication and proposes universal models for their automatic transcription to improve NLP understanding.

Contribution

It introduces a novel approach for integrating non-verbal cues into NLP by developing universal gesture segmentation and transcription models.

Findings

Highlights limitations of text-only models in capturing communication nuances

Proposes a flexible, efficient method for non-verbal cue transcription

Calls for community effort to develop and validate universal multimodal transcription methods

Abstract

This paper explores the critical but often overlooked role of non-verbal cues, including co-speech gestures and facial expressions, in human communication and their implications for Natural Language Processing (NLP). We argue that understanding human communication requires a more holistic approach that goes beyond textual or spoken words to include non-verbal elements. Borrowing from advances in sign language processing, we propose the development of universal automatic gesture segmentation and transcription models to transcribe these non-verbal cues into textual form. Such a methodology aims to bridge the blind spots in spoken language understanding, enhancing the scope and applicability of NLP models. Through motivating examples, we demonstrate the limitations of relying solely on text-based models. We propose a computationally efficient and flexible approach for incorporating non-verbal cues, which can seamlessly integrate with existing NLP pipelines. We conclude by calling upon the research community to contribute to the development of universal transcription methods and to validate their effectiveness in capturing the complexities of real-world, multi-modal interactions.