The Spatial Semantics of Iconic Gesture
Andy L\"ucking, Alexander Henlein, Alexander Mehler

TL;DR
This paper proposes a spatial gesture semantics framework that separates linguistic and visual meanings, modeling iconic gestures through vector translations, spatial evaluations, and linguistic classifications to better understand their role in visual communication.
Contribution
It introduces a novel three-level semantic model for iconic gestures, integrating kinematic, spatial, and linguistic aspects within a unified framework.
Findings
Iconic gestures can be modeled as vector sequences derived from kinematic data.
Spatial transformations like rotation and scaling are crucial for interpreting gestures.
Informational evaluation enables gestures to interact with verbal content in a quasi-linguistic manner.
Abstract
The current multimodal turn in linguistic theory leaves a crucial question unanswered: what is the meaning of iconic gestures, and how does it compose with speech meaning? We argue for a separation of linguistic and visual levels of meaning and introduce a spatial gesture semantics that closes this gap. Iconicity is differentiated into three aspects: Firstly, an interpretation of the form of a gesture in terms of a translation from kinematic gesture annotations into vector sequences (iconic model). Secondly, a truth-functional evaluation of the iconic model within spatially extended domains (embedding). Since a simple embedding is too strong, we identify a number of transformations that can be applied to iconic models, namely rotation, scaling, perspective fixation, and quotation of handshape. Thirdly, the linguistic description or classification of an iconic model (informational…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHand Gesture Recognition Systems · Hearing Impairment and Communication
