Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Models

TL;DR

This paper introduces a model merging technique that enhances zero-shot generalization in bioacoustic foundation models, significantly improving their ability to classify unseen species without sacrificing domain-specific performance.

Contribution

The authors propose a simple model merging strategy that interpolates a fine-tuned bioacoustic model with its base, boosting zero-shot generalization and instruction-following capabilities.

Findings

Over 200% improvement in zero-shot classification of unseen species

Merged model maintains high domain-specific accuracy

Achieves new state-of-the-art in bioacoustic zero-shot tasks

Abstract

Foundation models capable of generalizing across species and tasks represent a promising new frontier in bioacoustics, with NatureLM being one of the most prominent examples. While its domain-specific fine-tuning yields strong performance on bioacoustic benchmarks, we observe that it also introduces trade-offs in instruction-following flexibility. For instance, NatureLM achieves high accuracy when prompted for either the common or scientific name individually, but its accuracy drops significantly when both are requested in a single prompt. We address this by applying a simple model merging strategy that interpolates NatureLM with its base language model, recovering instruction-following capabilities with minimal loss of domain expertise. Finally, we show that the merged model exhibits markedly stronger zero-shot generalization, achieving over a 200% relative improvement and setting a new state-of-the-art in closed-set zero-shot classification of unseen species.