Guided Attention for Interpretable Motion Captioning

TL;DR

This paper presents a novel, interpretable motion captioning model that uses guided attention mechanisms to improve caption quality and provide insights into human motion, outperforming non-interpretable state-of-the-art systems.

Contribution

It introduces an attention-guided architecture for motion captioning that enhances interpretability and performance, with methods for guiding attention during training.

Findings

Improved captioning performance over state-of-the-art models.

Enhanced interpretability through attention guidance.

Ability to localize actions and identify body parts.

Abstract

Diverse and extensive work has recently been conducted on text-conditioned human motion generation. However, progress in the reverse direction, motion captioning, has seen less comparable advancement. In this paper, we introduce a novel architecture design that enhances text generation quality by emphasizing interpretability through spatio-temporal and adaptive attention mechanisms. To encourage human-like reasoning, we propose methods for guiding attention during training, emphasizing relevant skeleton areas over time and distinguishing motion-related words. We discuss and quantify our model's interpretability using relevant histograms and density distributions. Furthermore, we leverage interpretability to derive fine-grained information about human motion, including action localization, body part identification, and the distinction of motion-related words. Finally, we discuss the transferability of our approaches to other tasks. Our experiments demonstrate that attention guidance leads to interpretable captioning while enhancing performance compared to higher parameter-count, non-interpretable state-of-the-art systems. The code is available at: https://github.com/rd20karim/M2T-Interpretable.