RatBodyFormer: Rat Body Surface from Keypoints

TL;DR

This paper introduces RatBodyFormer, a novel method for reconstructing detailed rat body surfaces from sparse keypoints, enabling more nuanced analysis of rat behaviors beyond traditional pose estimation.

Contribution

It presents RatBodyFormer, a new neural network model that predicts rat body surface points from keypoints, and a multi-camera system, RatDome, with a large dataset for behavior analysis.

Findings

RatBodyFormer accurately reconstructs rat body surfaces from keypoints.

The method outperforms existing pose-based approaches in capturing subtle behaviors.

The framework enables advanced automated analysis of rat behaviors.

Abstract

Analyzing rat behavior lies at the heart of many scientific studies. Past methods for automated rodent modeling have focused on 3D pose estimation from keypoints, e.g., face and appendages. The pose, however, does not capture the rich body surface movement encoding the subtle rat behaviors like curling and stretching. The body surface lacks features that can be visually defined, evading these established keypoint-based methods. In this paper, we introduce the first method for reconstructing the rat body surface as a dense set of points by learning to predict it from the sparse keypoints that can be detected with past methods. Our method consists of two key contributions. The first is RatDome, a novel multi-camera system for rat behavior capture, and a large-scale dataset captured with it that consists of pairs of 3D keypoints and 3D body surface points. The second is RatBodyFormer, a novel network to transform detected keypoints to 3D body surface points. RatBodyFormer is agnostic to the exact locations of the 3D body surface points in the training data and is trained with masked-learning. We experimentally validate our framework with a number of real-world experiments. Our results collectively serve as a novel foundation for automated rat behavior analysis.