# vmTracking enables highly accurate multi-animal pose tracking in crowded environments

## TL;DR

vmTracking is a new method that improves multi-animal pose tracking in crowded environments by using virtual markers, reducing manual work and increasing accuracy.

## Contribution

vmTracking introduces virtual markers for individual identification, enabling accurate tracking in crowded and occluded scenarios.

## Key findings

- vmTracking minimizes manual corrections and annotation frames needed for training.
- Experiments confirmed vmTracking's effectiveness in tracking mice, fish, and human dancers.
- The method enhances precision and reliability in analyzing complex social behaviors.

## Abstract

In multi-animal tracking, addressing occlusion and crowding is crucial for accurate behavioral analysis. However, in situations where occlusion and crowding generate complex interactions, achieving accurate pose tracking remains challenging. Therefore, we introduced virtual marker tracking (vmTracking), which uses virtual markers for individual identification. Virtual markers are labels derived from conventional markerless multi-animal tracking tools, such as multi-animal DeepLabCut (maDLC) and Social LEAP Estimates Animal Poses (SLEAP). Unlike physical markers, virtual markers exist only within the video and attribute features to individuals, enabling consistent identification throughout the entire video while keeping the animals markerless in reality. Using these markers as cues, annotations were applied to multi-animal videos, and tracking was conducted with single-animal DeepLabCut (saDLC) and SLEAP’s single-animal method. vmTracking minimized manual corrections and annotation frames needed for training, efficiently tackling occlusion and crowding. Experiments tracking multiple mice, fish, and human dancers confirmed vmTracking’s variability and applicability. These findings could enhance the precision and reliability of tracking methods used in the analysis of complex naturalistic and social behaviors in animals, providing a simpler yet more effective solution.

Despite advancements in markerless multi-animal pose tracking tools, overcoming occlusion and crowding remains a challenge. This study develops a two-step method called vmTracking that enables highly accurate pose tracking of multiple animals in crowded environments.