Comparison of self-supervised in-domain and supervised out-domain transfer learning for bird species recognition

TL;DR

This paper compares the effectiveness of in-domain self-supervised and out-domain supervised transfer learning methods for bird species recognition, highlighting the benefits of in-domain models trained with VICReg.

Contribution

It provides an empirical comparison showing the advantages of in-domain self-supervised models over out-domain supervised models for bird species recognition.

Findings

In-domain self-supervised models outperform out-domain supervised models.

VICReg-based in-domain pre-training improves bird species recognition accuracy.

Self-supervised pre-training on domain-specific data is beneficial for data-scarce tasks.

Abstract

Transferring the weights of a pre-trained model to assist another task has become a crucial part of modern deep learning, particularly in data-scarce scenarios. Pre-training refers to the initial step of training models outside the current task of interest, typically on another dataset. It can be done via supervised models using human-annotated datasets or self-supervised models trained on unlabeled datasets. In both cases, many pre-trained models are available to fine-tune for the task of interest. Interestingly, research has shown that pre-trained models from ImageNet can be helpful for audio tasks despite being trained on image datasets. Hence, it's unclear whether in-domain models would be advantageous compared to competent out-domain models, such as convolutional neural networks from ImageNet. Our experiments will demonstrate the usefulness of in-domain models and datasets for bird species recognition by leveraging VICReg, a recent and powerful self-supervised method.