Seeing Sounds, Hearing Shapes: a gamified study to evaluate sound-sketches
Sebastian L\"obbers, Gy\"orgy Fazekas

TL;DR
This study explores how people use free-form sketches to communicate sound characteristics, revealing that while basic sound recognition is feasible, encoding detailed timbral nuances remains challenging.
Contribution
It introduces a gamified online methodology to evaluate sound-sketch associations and provides empirical data on the effectiveness of visual sketches in representing sound qualities.
Findings
Participants recognized sounds better than chance.
Encoding detailed timbral differences in sketches is difficult.
The method offers a new way to study cross-modal sound-shape associations.
Abstract
Sound-shape associations, a subset of cross-modal associations between the auditory and visual domain, have been studied mainly in the context of matching a set of purposefully crafted shapes to sounds. Recent studies have explored how humans represent sound through free-form sketching and how a graphical sketch input could be used for sound production. In this paper, the potential of communicating sound characteristics through these free-form sketches is investigated in a gamified study that was conducted with eighty-two participants at two online exhibition events. The results show that participants managed to recognise sounds at a higher rate than the random baseline would suggest, however it appeared difficult to visually encode nuanced timbral differences.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultisensory perception and integration · Music Technology and Sound Studies
