MLP: Motion Label Prior for Temporal Sentence Localization in Untrimmed 3D Human Motions

TL;DR

This paper introduces a novel label-prior-assisted training method for temporal sentence localization in 3D human motions, significantly improving accuracy by incorporating prior knowledge and refined predictions.

Contribution

It proposes two innovative label-prior-assisted training schemes specifically designed for 3D human motion localization, enhancing prediction accuracy over existing methods.

Findings

Achieved 44.13% recall at IoU@0.7 on BABEL dataset.

Attained 71.17% recall on HumanML3D (Restore) dataset.

Demonstrated effectiveness in corpus-level moment retrieval.

Abstract

In this paper, we address the unexplored question of temporal sentence localization in human motions (TSLM), aiming to locate a target moment from a 3D human motion that semantically corresponds to a text query. Considering that 3D human motions are captured using specialized motion capture devices, motions with only a few joints lack complex scene information like objects and lighting. Due to this character, motion data has low contextual richness and semantic ambiguity between frames, which limits the accuracy of predictions made by current video localization frameworks extended to TSLM to only a rough level. To refine this, we devise two novel label-prior-assisted training schemes: one embed prior knowledge of foreground and background to highlight the localization chances of target moments, and the other forces the originally rough predictions to overlap with the more accurate predictions obtained from the flipped start/end prior label sequences during recovery training. We show that injecting label-prior knowledge into the model is crucial for improving performance at high IoU. In our constructed TSLM benchmark, our model termed MLP achieves a recall of 44.13 at IoU@0.7 on the BABEL dataset and 71.17 on HumanML3D (Restore), outperforming prior works. Finally, we showcase the potential of our approach in corpus-level moment retrieval. Our source code is openly accessible at https://github.com/eanson023/mlp.