METEOR Guided Divergence for Video Captioning

TL;DR

This paper introduces a novel audiovisual video captioning approach using a reward-guided KL divergence and a Bi-Modal Hierarchical Reinforcement Learning Transformer to improve long-term dependency modeling and sentence quality.

Contribution

It proposes a new training method and a hierarchical transformer architecture that better captures temporal dependencies and audiovisual information for video captioning.

Findings

Achieved BLEU3 score of 4.91, BLEU4 score of 2.23, and METEOR score of 10.80 on ActivityNet Captions.

Demonstrated improved caption quality with content completeness and grammatical correctness.

Provided publicly available code and models for further research.

Abstract

Automatic video captioning aims for a holistic visual scene understanding. It requires a mechanism for capturing temporal context in video frames and the ability to comprehend the actions and associations of objects in a given timeframe. Such a system should additionally learn to abstract video sequences into sensible representations as well as to generate natural written language. While the majority of captioning models focus solely on the visual inputs, little attention has been paid to the audiovisual modality. To tackle this issue, we propose a novel two-fold approach. First, we implement a reward-guided KL Divergence to train a video captioning model which is resilient towards token permutations. Second, we utilise a Bi-Modal Hierarchical Reinforcement Learning (BMHRL) Transformer architecture to capture long-term temporal dependencies of the input data as a foundation for our hierarchical captioning module. Using our BMHRL, we show the suitability of the HRL agent in the generation of content-complete and grammatically sound sentences by achieving $4.91$, $2.23$, and $10.80$ in BLEU3, BLEU4, and METEOR scores, respectively on the ActivityNet Captions dataset. Finally, we make our BMHRL framework and trained models publicly available for users and developers at https://github.com/d-rothen/bmhrl.