Gesture Agreement Assessment Using Description Vectors

TL;DR

This paper introduces a new method using description vectors and a soft agreement rate metric to better assess consensus in gesture elicitation studies, demonstrated through a neurosurgeon study.

Contribution

It presents a novel partial similarity-based agreement metric (SAR) and a vector representation for gestures, improving agreement measurement over existing methods.

Findings

SAR metric yields 2.64 times higher agreement scores than existing metrics.

The methodology is validated through a gesture elicitation study with neurosurgeons.

SAR identifies the most agreed-upon gestures and descriptors, aiding design decisions.

Abstract

Participatory design is a popular design technique that involves the end users in the early stages of the design process to obtain user-friendly gestural interfaces. Guessability studies followed by agreement analyses are often used to elicit and comprehend the preferences (or gestures/proposals) of the participants. Previous approaches to assess agreement, grouped the gestures into equivalence classes and ignored the integral properties that are shared between them. In this work, we represent the gestures using binary description vectors to allow them to be partially similar. In this context, we introduce a new metric referred to as soft agreement rate (SAR) to quantify the level of consensus between the participants. In addition, we performed computational experiments to study the behavior of our partial agreement formula and mathematically show that existing agreement metrics are a special case of our approach. Our methodology was evaluated through a gesture elicitation study conducted with a group of neurosurgeons. Nevertheless, our formulation can be applied to any other user-elicitation study. Results show that the level of agreement obtained by SAR metric is 2.64 times higher than the existing metrics. In addition to the mostly agreed gesture, SAR formulation also provides the mostly agreed descriptors which can potentially help the designers to come up with a final gesture set.