Integrating Temporal and Spatial Attentions for VATEX Video Captioning Challenge 2019

TL;DR

This paper introduces a model for video captioning that combines temporal and spatial attention mechanisms, achieving high performance in the VATEX challenge by effectively capturing actions and objects.

Contribution

The novel integration of temporal and spatial attentions with late fusion for improved video captioning performance.

Findings

Achieved 73.4 CIDEr score on the VATEX test set

Ranked second in the VATEX 2019 challenge

Significantly outperformed baseline models

Abstract

This notebook paper presents our model in the VATEX video captioning challenge. In order to capture multi-level aspects in the video, we propose to integrate both temporal and spatial attentions for video captioning. The temporal attentive module focuses on global action movements while spatial attentive module enables to describe more fine-grained objects. Considering these two types of attentive modules are complementary, we thus fuse them via a late fusion strategy. The proposed model significantly outperforms baselines and achieves 73.4 CIDEr score on the testing set which ranks the second place at the VATEX video captioning challenge leaderboard 2019.