Navigating an Ocean of Video Data: Deep Learning for Humpback Whale Classification in YouTube Videos

TL;DR

This paper demonstrates that deep learning models can effectively classify YouTube videos to identify humpback whale encounters, enabling scalable analysis of social media data for biodiversity research.

Contribution

It introduces a CNN-RNN architecture pretrained on ImageNet for classifying whale videos, achieving high accuracy and F1 scores, thus automating social media data analysis for ecological studies.

Findings

Achieved 85.7% average accuracy in classification.

F1 scores of 84.7% (irrelevant) and 86.6% (relevant).

Deep learning enables efficient social media data use for biodiversity.

Abstract

Image analysis technologies empowered by artificial intelligence (AI) have proved images and videos to be an opportune source of data to learn about humpback whale (Megaptera novaeangliae) population sizes and dynamics. With the advent of social media, platforms such as YouTube present an abundance of video data across spatiotemporal contexts documenting humpback whale encounters from users worldwide. In our work, we focus on automating the classification of YouTube videos as relevant or irrelevant based on whether they document a true humpback whale encounter or not via deep learning. We use a CNN-RNN architecture pretrained on the ImageNet dataset for classification of YouTube videos as relevant or irrelevant. We achieve an average 85.7% accuracy, and 84.7% (irrelevant)/ 86.6% (relevant) F1 scores using five-fold cross validation for evaluation on the dataset. We show that deep learning can be used as a time-efficient step to make social media a viable source of image and video data for biodiversity assessments.