Voicing Uncertainty: How Speech, Text, and Visualizations Influence Decisions with Data Uncertainty

TL;DR

This study examines how different modalities of presenting data uncertainty—visual, text, and speech—affect decision-making, confidence, and trust, revealing that speech can increase risky choices and accents influence trust levels.

Contribution

It provides new insights into how multimodal uncertainty communication impacts user decisions and trust, highlighting the role of acoustic features and accents.

Findings

Speech representations increase risky decisions.

Text representations lower confidence.

Accent influences trust ratings.

Abstract

Understanding and communicating data uncertainty is crucial for informed decision-making across various domains, including finance, healthcare, and public policy. This study investigates the impact of gender and acoustic variables on decision-making, confidence, and trust through a crowdsourced experiment. We compared visualization-only representations of uncertainty to text-forward and speech-forward bimodal representations, including multiple synthetic voices across gender. Speech-forward representations led to an increase in risky decisions, and text-forward representations led to lower confidence. Contrary to prior work, speech-forward forecasts did not receive higher ratings of trust. Higher normalized pitch led to a slight increase in decision confidence, but other voice characteristics had minimal impact on decisions and trust. An exploratory analysis of accented speech showed consistent results with the main experiment and additionally indicated lower trust ratings for information presented in Indian and Kenyan accents. The results underscore the importance of considering acoustic and contextual factors in presentation of data uncertainty.