Seeing Sounds, Hearing Shapes: a gamified study to evaluate sound-sketches

TL;DR

This study explores how people use free-form sketches to communicate sound characteristics, revealing that while basic sound recognition is feasible, encoding detailed timbral nuances remains challenging.

Contribution

It introduces a gamified online methodology to evaluate sound-sketch associations and provides empirical data on the effectiveness of visual sketches in representing sound qualities.

Findings

Participants recognized sounds better than chance.

Encoding detailed timbral differences in sketches is difficult.

The method offers a new way to study cross-modal sound-shape associations.

Abstract

Sound-shape associations, a subset of cross-modal associations between the auditory and visual domain, have been studied mainly in the context of matching a set of purposefully crafted shapes to sounds. Recent studies have explored how humans represent sound through free-form sketching and how a graphical sketch input could be used for sound production. In this paper, the potential of communicating sound characteristics through these free-form sketches is investigated in a gamified study that was conducted with eighty-two participants at two online exhibition events. The results show that participants managed to recognise sounds at a higher rate than the random baseline would suggest, however it appeared difficult to visually encode nuanced timbral differences.