Imperial College London Submission to VATEX Video Captioning Task

TL;DR

This paper presents the Imperial College London's approach to the VATEX video captioning challenge, exploring sequence models and different conditioning strategies to improve captioning performance.

Contribution

It investigates the impact of conditioning sequence models on visual entity predictions versus pooled features for video captioning.

Findings

Entity-based conditioning outperforms pooled features.

Baseline models achieved competitive scores.

Conditional models are close in performance to sequence-to-sequence baselines.

Abstract

This paper describes the Imperial College London team's submission to the 2019' VATEX video captioning challenge, where we first explore two sequence-to-sequence models, namely a recurrent (GRU) model and a transformer model, which generate captions from the I3D action features. We then investigate the effect of dropping the encoder and the attention mechanism and instead conditioning the GRU decoder over two different vectorial representations: (i) a max-pooled action feature vector and (ii) the output of a multi-label classifier trained to predict visual entities from the action features. Our baselines achieved scores comparable to the official baseline. Conditioning over entity predictions performed substantially better than conditioning on the max-pooled feature vector, and only marginally worse than the GRU-based sequence-to-sequence baseline.