The "Seen but Unnoticed" Vocabulary of Natural Touch: Revolutionizing Direct Interaction with Our Devices and One Another (UIST 2021 Vision)

TL;DR

This paper advocates for expanding touch interaction to include natural human behaviors, contextual environments, and multi-device, multi-modal interactions to revolutionize personal and collaborative device use.

Contribution

It introduces a comprehensive vision for evolving touch interaction beyond traditional screens to encompass natural behaviors, multi-device ecosystems, and collaborative activities.

Findings

Existing touch interactions are limited and overlook natural behaviors.

A broader, context-aware approach can enhance individual and collaborative interactions.

Designing for multi-device, multi-modal touch can revolutionize user experiences.

Abstract

This UIST Vision argues that "touch" input and interaction remains in its infancy when viewed in context of the seen but unnoticed vocabulary of natural human behaviors, activity, and environments that surround direct interaction with displays. Unlike status-quo touch interaction -- a shadowplay of fingers on a single screen -- I argue that our perspective of direct interaction should encompass the full rich context of individual use (whether via touch, sensors, or in combination with other modalities), as well as collaborative activity where people are engaged in local (co-located), remote (tele-present), and hybrid work. We can further view touch through the lens of the "Society of Devices," where each person's activities span many complementary, oft-distinct devices that offer the right task affordance (input modality, screen size, aspect ratio, or simply a distinct surface with dedicated purpose) at the right place and time. While many hints of this vision already exist (see references), I speculate that a comprehensive program of research to systematically inventory, sense, and design interactions around such human behaviors and activities -- and that fully embrace touch as a multi-modal, multi-sensor, multi-user, and multi-device construct -- could revolutionize both individual and collaborative interaction with technology.