Dual-stream spatiotemporal networks with feature sharing for monitoring animals in the home cage

TL;DR

This paper introduces a dual-stream spatiotemporal neural network with feature sharing for improved mouse behavior classification in home cages, demonstrating enhanced accuracy over traditional models and potential for broader activity analysis.

Contribution

The paper proposes a novel feature sharing mechanism in dual-stream networks, significantly improving behavioral classification accuracy in animal monitoring tasks.

Findings

Feature sharing improves model performance consistently.

Achieved 86.47% accuracy with ensemble models.

Models generalize to other animal and human activity datasets.

Abstract

This paper presents a spatiotemporal deep learning approach for mouse behavioural classification in the home-cage. Using a series of dual-stream architectures with assorted modifications to increase performance, we introduce a novel feature sharing approach that jointly processes the streams at regular intervals throughout the network. To investigate the efficacy of this approach, models were evaluated by dissociating the streams and training/testing in the same rigorous manner as the main classifiers. Using an annotated, publicly available dataset of a singly-housed mice, we achieve prediction accuracy of 86.47% using an ensemble of a Inception-based network and an attention-based network, both of which utilize this feature sharing. We also demonstrate through ablation studies that for all models, the feature-sharing architectures consistently perform better than conventional ones having separate streams. The best performing models were further evaluated on other activity datasets, both mouse and human. Future work will investigate the effectiveness of feature sharing to behavioural classification in the unsupervised anomaly detection domain.