Increasing trust in new data sources: crowdsourcing image classification for ecology

TL;DR

This paper demonstrates that citizen science data, when analyzed with Bayesian models that account for participant ability and task difficulty, can reliably inform complex ecological research, exemplified by coral classification in the Great Barrier Reef.

Contribution

It introduces a Bayesian item response model for citizen science data that estimates participant ability, clusters participants, and improves ecological data analysis accuracy.

Findings

Bayesian model outperforms traditional majority vote methods.

Participants' abilities improve with more classification opportunities.

Weighted consensus enhances performance on difficult tasks.

Abstract

Crowdsourcing methods facilitate the production of scientific information by non-experts. This form of citizen science (CS) is becoming a key source of complementary data in many fields to inform data-driven decisions and study challenging problems. However, concerns about the validity of these data often constrain their utility. In this paper, we focus on the use of citizen science data in addressing complex challenges in environmental conservation. We consider this issue from three perspectives. First, we present a literature scan of papers that have employed Bayesian models with citizen science in ecology. Second, we compare several popular majority vote algorithms and introduce a Bayesian item response model that estimates and accounts for participants' abilities after adjusting for the difficulty of the images they have classified. The model also enables participants to be clustered into groups based on ability. Third, we apply the model in a case study involving the classification of corals from underwater images from the Great Barrier Reef, Australia. We show that the model achieved superior results in general and, for difficult tasks, a weighted consensus method that uses only groups of experts and experienced participants produced better performance measures. Moreover, we found that participants learn as they have more classification opportunities, which substantially increases their abilities over time. Overall, the paper demonstrates the feasibility of CS for answering complex and challenging ecological questions when these data are appropriately analysed. This serves as motivation for future work to increase the efficacy and trustworthiness of this emerging source of data.