Communicating Data Quality in On-Demand Curation

TL;DR

This paper evaluates how different visual representations of data uncertainty affect user interpretation in on-demand curation systems, emphasizing the need for effective UI guidelines.

Contribution

It presents a user study comparing four uncertainty representations, highlighting their impact on user understanding and reaction, and proposes initial UI design guidelines.

Findings

No significant difference in interpretation time across representations

Different uncertainty displays influence user perception and response

Establishes the need for UI guidelines for conveying data uncertainty

Abstract

On-demand curation (ODC) tools like Paygo, KATARA, and Mimir allow users to defer expensive curation effort until it is necessary. In contrast to classical databases that do not respond to queries over potentially erroneous data, ODC systems instead answer with guesses or approximations. The quality and scope of these guesses may vary and it is critical that an ODC system be able to communicate this information to an end-user. The central contribution of this paper is a preliminary user study evaluating the cognitive burden and expressiveness of four representations of "attribute-level" uncertainty. The study shows (1) insignificant differences in time taken for users to interpret the four types of uncertainty tested, and (2) that different presentations of uncertainty change the way people interpret and react to data. Ultimately, we show that a set of UI design guidelines and best practices for conveying uncertainty will be necessary for ODC tools to be effective. This paper represents the first step towards establishing such guidelines.