Spatial validation of acoustic individual identification models without ground truths: a case study with the cao-vit gibbon population

TL;DR

This paper introduces a new method to evaluate individual identification models for gibbons using spatial data, without needing labeled ground truth data.

Contribution

A novel framework for evaluating AIID models using territoriality assumptions and spatial data, without ground truth labels.

Findings

The method reliably estimates model accuracy with a root mean square error of 0.05 for models under 30% error rate.

Specific flaws in performance estimation are linked to types of AIID errors.

The approach is demonstrated on the critically endangered cao-vit gibbon population.

Abstract

Technological progress has made bioacoustics an important tool for research in the ecology and behaviour of sound producing animals. Using an array of synchronised autonomous recorders, we can localise vocalising animals, and for certain species, computational models can acoustically identify individuals (AIID). Knowing both the precise location and identity of vocalising animals enables a more detailed interpretation of long-term bioacoustic data, but assessing the reliability of AIID models is often difficult, especially for populations that evolve over time. Annotated ground truth labels in test sets are commonly used, but they are often limited in size, and there can be a mismatch with the application data (for instance in case of a change in recording system). Here, we formalise a methodology to evaluate AIID models based on localised predictions, thus bypassing the need for ground truth labels. We demonstrate it on a case study with the critically endangered cao-vit gibbons (Nomascus nasutus). Using deep-learning, we develop an AIID model for male cao-vit gibbons. Then, we estimate its performance without any ground truths, using a new framework that relies on assumptions of territoriality. Empirical tests with simulated data show that this approach to no-ground-truth AIID evaluation is fairly reliable (0.05 of root mean square error between estimated and real accuracy for models with less than 30% of error rate), and specific flaws of performance estimation are described according to specific types of AIID errors. With this article, we demonstrate how spatialised data might help in the evaluation of AIID models for territorial species, both theoretically, and in practice with the cao-vit gibbon population.

Figures