The Birds Need Attention Too: Analysing usage of Self Attention in identifying bird calls in soundscapes

TL;DR

This paper explores the use of self-attention mechanisms, specifically transformer models, for automatic bird call recognition in soundscapes, demonstrating superior performance over traditional convolutional neural networks.

Contribution

It introduces a pre-trained Attention-based Spectrogram Transformer for bird call identification and compares its effectiveness against convolutional models on BirdCLEF datasets.

Findings

Transformer models outperform convolutional models in bird call recognition

Self-attention mechanisms improve accuracy in diverse soundscape environments

Validated results across BirdCLEF 2021 and 2022 datasets

Abstract

Birds are vital parts of ecosystems across the world and are an excellent measure of the quality of life on earth. Many bird species are endangered while others are already extinct. Ecological efforts in understanding and monitoring bird populations are important to conserve their habitat and species, but this mostly relies on manual methods in rough terrains. Recent advances in Machine Learning and Deep Learning have made automatic bird recognition in diverse environments possible. Birdcall recognition till now has been performed using convolutional neural networks. In this work, we try and understand how self-attention can aid in this endeavor. With that we build an pre-trained Attention-based Spectrogram Transformer baseline for BirdCLEF 2022 and compare the results against the pre-trained Convolution-based baseline. Our results show that the transformer models outperformed the convolutional model and we further validate our results by building baselines and analyzing the results for the previous year BirdCLEF 2021 challenge. Source code available at https://github.com/ck090/BirdCLEF-22