Human Factors Influencing Visual Statistical Inference

TL;DR

This paper investigates how various human factors like skills, demographics, and experimental setup influence the effectiveness of visual statistical inference, highlighting the variability and robustness of human judgment in data analysis.

Contribution

It provides an empirical assessment of how individual differences and experimental conditions affect the accuracy of visual inference using lineup plots.

Findings

Individual skills vary substantially among observers.

Demographics have minimal impact on visual inference performance.

Placement of data plot does not influence inference accuracy.

Abstract

Visual statistical inference is a way to determine significance of patterns found while exploring data. It is dependent on the evaluation of a lineup, of a data plot among a sample of null plots, by human observers. Each individual is different in their cognitive psychology and judiciousness, which can affect the visual inference. The usual way to estimate the effectiveness of a statistical test is its power. The estimate of power of a lineup can be controlled by combining evaluations from multiple observers. Factors that may also affect the power of visual inference are the observers' demographics, visual skills, and experience, the sample of null plots taken from the null distribution, the position of the data plot in the lineup, and the signal strength in the data. This paper examines these factors. Results from multiple visual inference studies using Amazon's Mechanical Turk are examined to provide an assessment of these. The experiments suggest that individual skills vary substantially, but demographics do not have a huge effect on performance. There is evidence that a learning effect exists but only in that observers get faster with repeated evaluations, but not more often correct. The placement of data plot in the lineup does not affect the inference.