A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

TL;DR

This paper systematically compares various animal action segmentation algorithms across supervised, unsupervised, and semi-supervised paradigms, introduces a semi-supervised model, and finds supervised temporal convolutional networks perform best.

Contribution

It provides a comprehensive comparison of existing algorithms across multiple datasets and introduces a novel semi-supervised segmentation model.

Findings

Supervised temporal convolutional networks outperform other methods.

Adding temporal information improves segmentation accuracy.

Semi-supervised model bridges supervised and unsupervised approaches.

Abstract

Action segmentation of behavioral videos is the process of labeling each frame as belonging to one or more discrete classes, and is a crucial component of many studies that investigate animal behavior. A wide range of algorithms exist to automatically parse discrete animal behavior, encompassing supervised, unsupervised, and semi-supervised learning paradigms. These algorithms -- which include tree-based models, deep neural networks, and graphical models -- differ widely in their structure and assumptions on the data. Using four datasets spanning multiple species -- fly, mouse, and human -- we systematically study how the outputs of these various algorithms align with manually annotated behaviors of interest. Along the way, we introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information in the observations perform the best on our supervised metrics across all datasets.