Two Empirical Studies on Audiovisual Semiotics of Uncertainty

TL;DR

This paper investigates how audiovisual semiotics can improve the perception of uncertainty by analyzing user preferences and reaction times in two empirical studies, providing guidelines for effective audiovisual mappings.

Contribution

It offers the first empirical evaluation of audiovisual mappings for uncertainty, identifying preferred combinations and proposing guidelines for effective audiovisual representations.

Findings

Certain audiovisual pairs are preferred by users.

Preferred pairs are not always the most effective for representing uncertainty.

Guidelines for audiovisual mappings that enhance perception of uncertainty.

Abstract

There exists limited theoretical guidance on integrating visualization and sonification. In this paper, we address this gap by investigating audiovisual semiotics for uncertainty representation: joining uncertainty visualization and sonification to combine audiovisual channels for enhancing users' perception of uncertainty. We conducted two preregistered crowd-sourced user studies. First, we assessed suitable audio/visual pairs. Then, we investigated audiovisual mappings of uncertainty. Here, we use probability as it is an easily communicated aspect of uncertainty. We analyzed the participants' preferences and reaction times in both user studies. Additionally, we explored the strategies employed by participants through qualitative analysis. Our results reveal audiovisual mappings that lead to particularly strong preferences and low reaction times. Furthermore, we found that preferred audio/visual pairs are not necessarily suitable audiovisual mappings of uncertainty. For example, while pitch paired with brightness was preferred as a pair, it was not well suited as a mapping for uncertainty. We recommend audiovisual mappings of uncertainty that lead to low reaction times and high preferences in both user studies. This paper presents guidelines to anyone seeking to employ audiovisual representations for uncertainty, contributing to enhancing the perception of uncertainty.