Learning Keypoints for Multi-Agent Behavior Analysis using Self-Supervision

TL;DR

This paper introduces B-KinD-multi, a self-supervised method leveraging pre-trained segmentation models to automatically discover keypoints in multi-agent videos, improving behavior analysis across various species without manual annotations.

Contribution

The novel approach uses pre-trained segmentation to enhance keypoint discovery in multi-agent videos, addressing challenges of existing methods in complex scenarios.

Findings

Improved keypoint regression accuracy.

Enhanced behavioral classification performance.

Good generalization across multiple species.

Abstract

The study of social interactions and collective behaviors through multi-agent video analysis is crucial in biology. While self-supervised keypoint discovery has emerged as a promising solution to reduce the need for manual keypoint annotations, existing methods often struggle with videos containing multiple interacting agents, especially those of the same species and color. To address this, we introduce B-KinD-multi, a novel approach that leverages pre-trained video segmentation models to guide keypoint discovery in multi-agent scenarios. This eliminates the need for time-consuming manual annotations on new experimental settings and organisms. Extensive evaluations demonstrate improved keypoint regression and downstream behavioral classification in videos of flies, mice, and rats. Furthermore, our method generalizes well to other species, including ants, bees, and humans, highlighting its potential for broad applications in automated keypoint annotation for multi-agent behavior analysis. Code available under: https://danielpkhalil.github.io/B-KinD-Multi