Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Tasks

TL;DR

This study explores how humans develop shared communication conventions through language and gestures during collaborative physical tasks, revealing how multimodal interactions evolve to improve efficiency and accuracy.

Contribution

It introduces a probabilistic model of multimodal convention formation and demonstrates how communication strategies adapt in physical collaboration tasks.

Findings

Participants became faster and more accurate over time.

Establishment of linguistic and gestural abstractions.

Use of cross-modal redundancy enhanced communication.

Abstract

A quintessential feature of human intelligence is the ability to create ad hoc conventions over time to achieve shared goals efficiently. We investigate how communication strategies evolve through repeated collaboration as people coordinate on shared procedural abstractions. To this end, we conducted an online unimodal study (n = 98) using natural language to probe abstraction hierarchies. In a follow-up lab study (n = 40), we examined how multimodal communication (speech and gestures) changed during physical collaboration. Pairs used augmented reality to isolate their partner's hand and voice; one participant viewed a 3D virtual tower and sent instructions to the other, who built the physical tower. Participants became faster and more accurate by establishing linguistic and gestural abstractions and using cross-modal redundancy to emphasize key changes from previous interactions. Based on these findings, we extend probabilistic models of convention formation to multimodal settings, capturing shifts in modality preferences. Our findings and model provide building blocks for designing convention-aware intelligent agents situated in the physical world.