Deep Multi-stream Network for Video-based Calving Sign Detection

TL;DR

This paper introduces a multi-stream deep neural network that integrates expert knowledge to detect calving signs from video, improving accuracy and interpretability over traditional end-to-end models for livestock management.

Contribution

The study presents a novel multi-stream network architecture that combines multiple calving-related features, enhancing detection performance and interpretability in practical livestock scenarios.

Findings

Multi-stream system outperformed end-to-end system in accuracy.

Significant reduction in detection errors with multi-stream architecture.

Mixture weights provided interpretability of model behavior.

Abstract

We have designed a deep multi-stream network for automatically detecting calving signs from video. Calving sign detection from a camera, which is a non-contact sensor, is expected to enable more efficient livestock management. As large-scale, well-developed data cannot generally be assumed when establishing calving detection systems, the basis for making the prediction needs to be presented to farmers during operation, so black-box modeling (also known as end-to-end modeling) is not appropriate. For practical operation of calving detection systems, the present study aims to incorporate expert knowledge into a deep neural network. To this end, we propose a multi-stream calving sign detection network in which multiple calving-related features are extracted from the corresponding feature extraction networks designed for each attribute with different characteristics, such as a cow's posture, rotation, and movement, known as calving signs, and are then integrated appropriately depending on the cow's situation. Experimental comparisons conducted using videos of 15 cows demonstrated that our multi-stream system yielded a significant improvement over the end-to-end system, and the multi-stream architecture significantly contributed to a reduction in detection errors. In addition, the distinctive mixture weights we observed helped provide interpretability of the system's behavior.