BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics

TL;DR

BIRB is a comprehensive benchmark designed to evaluate machine learning models' ability to retrieve bird vocalizations across diverse and realistic conditions, addressing the gap in existing artificial benchmarks.

Contribution

We introduce BIRB, a complex, realistic benchmark for bird sound retrieval, and propose a baseline system to evaluate model robustness and generalization in bioacoustics.

Findings

BIRB reveals significant challenges in model generalization across distribution shifts.

Representation learning with nearest-centroid search provides a strong baseline.

Analysis suggests directions for improving robustness in bioacoustic retrieval models.

Abstract

The ability for a machine learning model to cope with differences in training and deployment conditions--e.g. in the presence of distribution shift or the generalization to new classes altogether--is crucial for real-world use cases. However, most empirical work in this area has focused on the image domain with artificial benchmarks constructed to measure individual aspects of generalization. We present BIRB, a complex benchmark centered on the retrieval of bird vocalizations from passively-recorded datasets given focal recordings from a large citizen science corpus available for training. We propose a baseline system for this collection of tasks using representation learning and a nearest-centroid search. Our thorough empirical evaluation and analysis surfaces open research directions, suggesting that BIRB fills the need for a more realistic and complex benchmark to drive progress on robustness to distribution shifts and generalization of ML models.