SemiMultiPose: A Semi-supervised Multi-animal Pose Estimation Framework

TL;DR

SemiMultiPose introduces a semi-supervised framework that effectively utilizes unlabeled data to improve multi-animal pose estimation, especially in scenarios with limited labeled annotations, outperforming existing methods.

Contribution

It presents a novel semi-supervised architecture that leverages unlabeled frames' inherent structures to enhance multi-animal pose estimation performance.

Findings

Outperforms state-of-the-art baselines on three animal datasets.

Shows increased accuracy in sparsely-labeled data regimes.

Demonstrates effective use of unlabeled data in pose estimation.

Abstract

Multi-animal pose estimation is essential for studying animals' social behaviors in neuroscience and neuroethology. Advanced approaches have been proposed to support multi-animal estimation and achieve state-of-the-art performance. However, these models rarely exploit unlabeled data during training even though real world applications have exponentially more unlabeled frames than labeled frames. Manually adding dense annotations for a large number of images or videos is costly and labor-intensive, especially for multiple instances. Given these deficiencies, we propose a novel semi-supervised architecture for multi-animal pose estimation, leveraging the abundant structures pervasive in unlabeled frames in behavior videos to enhance training, which is critical for sparsely-labeled problems. The resulting algorithm will provide superior multi-animal pose estimation results on three animal experiments compared to the state-of-the-art baseline and exhibits more predictive power in sparsely-labeled data regimes.