A Study of Material Sonification in Touchscreen Devices

TL;DR

This paper explores using interactive sound to enhance the perception of material qualities on touchscreen devices, finding that audio cues increase immersion but do not significantly affect quality perception.

Contribution

It introduces a novel application translating tactile input into material sounds and evaluates its impact on perception and immersion.

Findings

Audio cues do not significantly alter perception of material qualities.

Sound increases user immersion during interaction.

The system provides a new multisensory communication method.

Abstract

Even in the digital age, designers largely rely on physical material samples to illustrate their products, as existing visual representations fail to sufficiently reproduce the look and feel of real world materials. Here, we investigate the use of interactive material sonification as an additional sensory modality for communicating well-established material qualities like softness, pleasantness or value. We developed a custom application for touchscreen devices that receives tactile input and translate it into material rubbing sound using granular synthesis. We used this system to perform a psychophysical study, in which the ability of the user to rate subjective material qualities is evaluated, with the actual material samples serving as reference stimulus. Our experimental results indicate that the considered audio cues do not significantly contribute to the perception of material qualities but are able to increase the level of immersion when interacting with digital samples.