Transformer with Controlled Attention for Synchronous Motion Captioning

TL;DR

This paper introduces a Transformer-based model with controlled attention mechanisms for synchronous motion captioning, enabling interpretable, time-aligned text generation synchronized with human motion sequences, and demonstrates superior results on benchmark datasets.

Contribution

The paper proposes a novel attention control method within Transformers for synchronized motion captioning, enhancing interpretability and temporal alignment.

Findings

Achieved state-of-the-art performance on KIT-ML and HumanML3D datasets.

Provided visualizations demonstrating synchronized motion and caption generation.

Introduced masking strategies to focus attention on important motion frames.

Abstract

In this paper, we address a challenging task, synchronous motion captioning, that aim to generate a language description synchronized with human motion sequences. This task pertains to numerous applications, such as aligned sign language transcription, unsupervised action segmentation and temporal grounding. Our method introduces mechanisms to control self- and cross-attention distributions of the Transformer, allowing interpretability and time-aligned text generation. We achieve this through masking strategies and structuring losses that push the model to maximize attention only on the most important frames contributing to the generation of a motion word. These constraints aim to prevent undesired mixing of information in attention maps and to provide a monotonic attention distribution across tokens. Thus, the cross attentions of tokens are used for progressive text generation in synchronization with human motion sequences. We demonstrate the superior performance of our approach through evaluation on the two available benchmark datasets, KIT-ML and HumanML3D. As visual evaluation is essential for this task, we provide a comprehensive set of animated visual illustrations in the code repository: https://github.com/rd20karim/Synch-Transformer.