APT-36K: A Large-scale Benchmark for Animal Pose Estimation and Tracking

TL;DR

APT-36K is the first large-scale benchmark dataset for animal pose estimation and tracking, enabling comprehensive evaluation and advancing research in video-based animal behavior analysis.

Contribution

This paper introduces APT-36K, a large-scale, high-quality dataset for animal pose estimation and tracking, filling a critical gap in existing datasets and providing a new resource for the community.

Findings

Benchmark results reveal challenges in cross-species generalization.

Supervised models perform well within species but struggle across species.

The dataset facilitates evaluation of both pose estimation and tracking in videos.

Abstract

Animal pose estimation and tracking (APT) is a fundamental task for detecting and tracking animal keypoints from a sequence of video frames. Previous animal-related datasets focus either on animal tracking or single-frame animal pose estimation, and never on both aspects. The lack of APT datasets hinders the development and evaluation of video-based animal pose estimation and tracking methods, limiting real-world applications, e.g., understanding animal behavior in wildlife conservation. To fill this gap, we make the first step and propose APT-36K, i.e., the first large-scale benchmark for animal pose estimation and tracking. Specifically, APT-36K consists of 2,400 video clips collected and filtered from 30 animal species with 15 frames for each video, resulting in 36,000 frames in total. After manual annotation and careful double-check, high-quality keypoint and tracking annotations are provided for all the animal instances. Based on APT-36K, we benchmark several representative models on the following three tracks: (1) supervised animal pose estimation on a single frame under intra- and inter-domain transfer learning settings, (2) inter-species domain generalization test for unseen animals, and (3) animal pose estimation with animal tracking. Based on the experimental results, we gain some empirical insights and show that APT-36K provides a valuable animal pose estimation and tracking benchmark, offering new challenges and opportunities for future research. The code and dataset will be made publicly available at https://github.com/pandorgan/APT-36K.