Variational Stacked Local Attention Networks for Diverse Video Captioning

TL;DR

This paper introduces VSLAN, a novel video captioning model that enhances feature interaction and diversity in generated captions through variational stacked local attention and multiple feature streams, outperforming existing methods.

Contribution

The paper proposes VSLAN, a new model that uses low-rank bilinear pooling and feature stacking to improve caption diversity and accuracy without explicit supervision.

Findings

VSLAN outperforms existing methods on CIDEr scores by 7.8% on MSVD and 4.5% on MSR-VTT.

VSLAN achieves competitive results in caption diversity metrics.

The model effectively captures fine-grained visual features for diverse caption generation.

Abstract

While describing Spatio-temporal events in natural language, video captioning models mostly rely on the encoder's latent visual representation. Recent progress on the encoder-decoder model attends encoder features mainly in linear interaction with the decoder. However, growing model complexity for visual data encourages more explicit feature interaction for fine-grained information, which is currently absent in the video captioning domain. Moreover, feature aggregations methods have been used to unveil richer visual representation, either by the concatenation or using a linear layer. Though feature sets for a video semantically overlap to some extent, these approaches result in objective mismatch and feature redundancy. In addition, diversity in captions is a fundamental component of expressing one event from several meaningful perspectives, currently missing in the temporal, i.e., video captioning domain. To this end, we propose Variational Stacked Local Attention Network (VSLAN), which exploits low-rank bilinear pooling for self-attentive feature interaction and stacking multiple video feature streams in a discount fashion. Each feature stack's learned attributes contribute to our proposed diversity encoding module, followed by the decoding query stage to facilitate end-to-end diverse and natural captions without any explicit supervision on attributes. We evaluate VSLAN on MSVD and MSR-VTT datasets in terms of syntax and diversity. The CIDEr score of VSLAN outperforms current off-the-shelf methods by $7.8\%$ on MSVD and $4.5\%$ on MSR-VTT, respectively. On the same datasets, VSLAN achieves competitive results in caption diversity metrics.