The Spatial Semantics of Iconic Gesture

TL;DR

This paper proposes a spatial gesture semantics framework that separates linguistic and visual meanings, modeling iconic gestures through vector translations, spatial evaluations, and linguistic classifications to better understand their role in visual communication.

Contribution

It introduces a novel three-level semantic model for iconic gestures, integrating kinematic, spatial, and linguistic aspects within a unified framework.

Findings

Iconic gestures can be modeled as vector sequences derived from kinematic data.

Spatial transformations like rotation and scaling are crucial for interpreting gestures.

Informational evaluation enables gestures to interact with verbal content in a quasi-linguistic manner.

Abstract

The current multimodal turn in linguistic theory leaves a crucial question unanswered: what is the meaning of iconic gestures, and how does it compose with speech meaning? We argue for a separation of linguistic and visual levels of meaning and introduce a spatial gesture semantics that closes this gap. Iconicity is differentiated into three aspects: Firstly, an interpretation of the form of a gesture in terms of a translation from kinematic gesture annotations into vector sequences (iconic model). Secondly, a truth-functional evaluation of the iconic model within spatially extended domains (embedding). Since a simple embedding is too strong, we identify a number of transformations that can be applied to iconic models, namely rotation, scaling, perspective fixation, and quotation of handshape. Thirdly, the linguistic description or classification of an iconic model (informational evaluation). Since the informational evaluation of an iconic gesture is a heuristic act, it needs a place in a semantic theory of visual communication. Informational evaluation lifts a gesture to a quasi-linguistic level that can interact with verbal content. This interaction is either vacuous, or regimented by usual lexicon-driven inferences discussed in dynamic semantic frameworks.