Is that a Duiker or Dik Dik Next to the Giraffe? Impacts of Uncertainty on Classification Efficiency in Citizen Science

TL;DR

This paper investigates how content complexity and image quality issues affect classification accuracy and consensus in citizen science projects, proposing content-based measures to improve quality control.

Contribution

It introduces a content categorization method based on aggregate classifications and analyzes how content complexity impacts classification efficiency.

Findings

Content complexity categories influence classification efficiency.

Image quality issues affect volunteers' confidence in classification.

Different content types may require tailored consensus measures.

Abstract

Quality control is an ongoing concern in citizen science that is often managed by replication to consensus in online tasks such as image classification. Numerous factors can lead to disagreement, including image quality problems, interface specifics, and the complexity of the content itself. We conducted trace ethnography with statistical and qualitative analyses of six Snapshot Safari projects to understand the content characteristics that can lead to uncertainty and low consensus. This study contributes content categorization based on aggregate classifications to characterize image complexity, with analysis that confirms that the categories impact classification efficiency, and an inductively generated set of additional image quality issues that also impact volunteers' ability to confidently classify content. The results suggest that different conceptualizations and measures of consensus may be needed for different types of content, and aggregate responses offer a way to identify content that needs different handling when complexity cannot be determined $a$ $priori$.