NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics

TL;DR

NatureLM-audio is a pioneering audio-language foundation model tailored for bioacoustics, enabling improved zero-shot classification and generalization across diverse animal vocalization tasks, thereby supporting conservation and biodiversity efforts.

Contribution

It introduces the first bioacoustics-specific audio-language foundation model trained on curated data, demonstrating transfer learning from speech and music to bioacoustics and establishing new state-of-the-art results.

Findings

Sets new state-of-the-art on bioacoustics tasks

Shows effective transfer from speech and music models

Achieves promising generalization to unseen species

Abstract

Large language models (LLMs) prompted with text and audio have achieved state-of-the-art performance across various auditory tasks, including speech, music, and general audio, showing emergent abilities on unseen tasks. However, their potential has yet to be fully demonstrated in bioacoustics tasks, such as detecting animal vocalizations in large recordings, classifying rare and endangered species, and labeling context and behavior -- tasks that are crucial for conservation, biodiversity monitoring, and animal behavior studies. In this work, we present NatureLM-audio, the first audio-language foundation model specifically designed for bioacoustics. Our training dataset consists of carefully curated text-audio pairs spanning bioacoustics, speech, and music, designed to address the field's limited availability of annotated data. We demonstrate successful transfer of learned representations from music and speech to bioacoustics, and our model shows promising generalization to unseen taxa and tasks. We evaluate NatureLM-audio on a novel benchmark (BEANS-Zero) and it sets a new state of the art on several bioacoustics tasks, including zero-shot classification of unseen species. To advance bioacoustics research, we release our model weights, benchmark data, and open-source the code for training and benchmark data generation and model training.