Analysis of Video Feature Learning in Two-Stream CNNs on the Example of Zebrafish Swim Bout Classification

TL;DR

This paper investigates how two-stream CNNs classify zebrafish swim bouts, using explainability techniques to understand their decision process, revealing differences from manual features and improving accuracy over previous SVM methods.

Contribution

It introduces the use of Deep Taylor Decomposition for CNN interpretability in zebrafish behavior classification and demonstrates how removing artifacts enhances model accuracy.

Findings

CNN focuses on tail trunk steadiness, unlike manual features.

Removing artifacts improves classification accuracy.

Best CNN achieves 96.32% accuracy, surpassing SVM by 6.12%.

Abstract

Semmelhack et al. (2014) have achieved high classification accuracy in distinguishing swim bouts of zebrafish using a Support Vector Machine (SVM). Convolutional Neural Networks (CNNs) have reached superior performance in various image recognition tasks over SVMs, but these powerful networks remain a black box. Reaching better transparency helps to build trust in their classifications and makes learned features interpretable to experts. Using a recently developed technique called Deep Taylor Decomposition, we generated heatmaps to highlight input regions of high relevance for predictions. We find that our CNN makes predictions by analyzing the steadiness of the tail's trunk, which markedly differs from the manually extracted features used by Semmelhack et al. (2014). We further uncovered that the network paid attention to experimental artifacts. Removing these artifacts ensured the validity of predictions. After correction, our best CNN beats the SVM by 6.12%, achieving a classification accuracy of 96.32%. Our work thus demonstrates the utility of AI explainability for CNNs.