Annotation aggregation of multi-label ecological datasets via Bayesian modeling

TL;DR

This paper introduces a Bayesian hierarchical model to effectively aggregate sparse, variable expert annotations in ecological datasets, enhancing bird species classification and uncertainty quantification in large-scale audio monitoring.

Contribution

It presents a novel Bayesian modeling method for combining expert annotations with varying accuracy, improving classification and providing performance scores.

Findings

Improved bird species classification accuracy.

Effective uncertainty quantification for expert labels.

Enhanced engagement through performance scoring.

Abstract

Ecological and conservation studies monitoring bird communities typically rely on species classification based on bird vocalizations. Historically, this has been based on expert volunteers going into the field and making lists of the bird species that they observe. Recently, machine learning algorithms have emerged that can accurately classify bird species based on audio recordings of their vocalizations. Such algorithms crucially rely on training data that are labeled by experts. Automated classification is challenging when multiple species are vocalizing simultaneously, there is background noise, and/or the bird is far from the microphone. In continuously monitoring different locations, the size of the audio data become immense and it is only possible for human experts to label a tiny proportion of the available data. In addition, experts can vary in their accuracy and breadth of knowledge about different species. This article focuses on the important problem of combining sparse expert annotations to improve bird species classification while providing uncertainty quantification. We additionally are interested in providing expert performance scores to increase their engagement and encourage improvements. We propose a Bayesian hierarchical modeling approach and evaluate this approach on a new community science platform developed in Finland.