Characterizing the impact of using features extracted from pre-trained models on the quality of video captioning sequence-to-sequence models

TL;DR

This paper investigates how features extracted from pre-trained models across multiple modalities influence the quality of video captioning in sequence-to-sequence models, highlighting the importance of multi-modal features.

Contribution

It characterizes the impact of various pre-trained features and their interactions on video captioning quality, demonstrating the benefits of multi-modal feature integration.

Findings

Multi-modal features significantly improve caption quality.

Interactions among features affect the final representation.

Including diverse features enhances encoder-decoder performance.

Abstract

The task of video captioning, that is, the automatic generation of sentences describing a sequence of actions in a video, has attracted an increasing attention recently. The complex and high-dimensional representation of video data makes it difficult for a typical encoder-decoder architectures to recognize relevant features and encode them in a proper format. Video data contains different modalities that can be recognized using a mix image, scene, action and audio features. In this paper, we characterize the different features affecting video descriptions and explore the interactions among these features and how they affect the final quality of a video representation. Building on existing encoder-decoder models that utilize limited range of video information, our comparisons show how the inclusion of multi-modal video features can make a significant effect on improving the quality of generated statements. The work is of special interest to scientists and practitioners who are using sequence-to-sequence models to generate video captions.