Prompt When the Animal is: Temporal Animal Behavior Grounding with Positional Recovery Training

TL;DR

This paper introduces Port, a novel training framework for temporal animal behavior grounding that uses positional prompts and dual-alignment to improve model focus on specific behaviors, demonstrating superior performance on the Animal Kingdom dataset.

Contribution

The paper proposes Port, a new training method with a Recovering branch and dual-alignment, specifically designed to handle the challenges of sparse and uniformly distributed animal behavior data.

Findings

Achieved IoU@0.3 of 38.52 on Animal Kingdom dataset.

Outperformed baseline models and was a top performer in ICME 2024 Grand Challenges.

Demonstrated effectiveness of positional prompts in temporal grounding tasks.

Abstract

Temporal grounding is crucial in multimodal learning, but it poses challenges when applied to animal behavior data due to the sparsity and uniform distribution of moments. To address these challenges, we propose a novel Positional Recovery Training framework (Port), which prompts the model with the start and end times of specific animal behaviors during training. Specifically, \port{} enhances the baseline model with a Recovering branch to reconstruct corrupted label sequences and align distributions via a Dual-alignment method. This allows the model to focus on specific temporal regions prompted by ground-truth information. Extensive experiments on the Animal Kingdom dataset demonstrate the effectiveness of \port{}, achieving an IoU@0.3 of 38.52. It emerges as one of the top performers in the sub-track of MMVRAC in ICME 2024 Grand Challenges.