Exploration of Visual Features and their weighted-additive fusion for Video Captioning

TL;DR

This paper presents WAFTM, a novel video captioning model that fuses visual features with weighted-additive methods and memory-augmented transformers, achieving improved performance on benchmark datasets.

Contribution

Introduction of WAFTM, a transformer-based video captioning model with memory and a new feature fusion technique that emphasizes significant visual representations.

Findings

Achieved CIDEr score of 92.4 on MSVD dataset.

Obtained METEOR score of 0.091 on ActivityNet Captions.

Demonstrated performance gains with Word-Piece Tokenization and REINFORCE.

Abstract

Video captioning is a popular task that challenges models to describe events in videos using natural language. In this work, we investigate the ability of various visual feature representations derived from state-of-the-art convolutional neural networks to capture high-level semantic context. We introduce the Weighted Additive Fusion Transformer with Memory Augmented Encoders (WAFTM), a captioning model that incorporates memory in a transformer encoder and uses a novel method, to fuse features, that ensures due importance is given to more significant representations. We illustrate a gain in performance realized by applying Word-Piece Tokenization and a popular REINFORCE algorithm. Finally, we benchmark our model on two datasets and obtain a CIDEr of 92.4 on MSVD and a METEOR of 0.091 on the ActivityNet Captions Dataset.