Co-Speech Gesture Detection through Multi-Phase Sequence Labeling

TL;DR

This paper presents a novel multi-phase sequence labeling framework for co-speech gesture detection, utilizing Transformer encoders and CRFs to better capture gesture dynamics over traditional binary classification methods.

Contribution

It introduces a new approach that models gesture phases as a sequence labeling problem, improving detection accuracy over existing binary classification methods.

Findings

Significant performance improvement over baseline models.

Transformer encoders enhance contextual understanding of gesture sequences.

Effective detection of gesture stroke phases in face-to-face dialogues.

Abstract

Gestures are integral components of face-to-face communication. They unfold over time, often following predictable movement phases of preparation, stroke, and retraction. Yet, the prevalent approach to automatic gesture detection treats the problem as binary classification, classifying a segment as either containing a gesture or not, thus failing to capture its inherently sequential and contextual nature. To address this, we introduce a novel framework that reframes the task as a multi-phase sequence labeling problem rather than binary classification. Our model processes sequences of skeletal movements over time windows, uses Transformer encoders to learn contextual embeddings, and leverages Conditional Random Fields to perform sequence labeling. We evaluate our proposal on a large dataset of diverse co-speech gestures in task-oriented face-to-face dialogues. The results consistently demonstrate that our method significantly outperforms strong baseline models in detecting gesture strokes. Furthermore, applying Transformer encoders to learn contextual embeddings from movement sequences substantially improves gesture unit detection. These results highlight our framework's capacity to capture the fine-grained dynamics of co-speech gesture phases, paving the way for more nuanced and accurate gesture detection and analysis.