The Role of the Input in Natural Language Video Description

TL;DR

This paper investigates how visual input transformations affect the performance of natural language video description systems, using augmented datasets and analysis to improve understanding and robustness in real-world scenarios.

Contribution

It introduces a comprehensive study on the impact of visual input processing on NLVD, including data augmentation and a new dataset version, MSVD-v2.

Findings

Visual transformations influence NLVD performance.

Augmented data improves model robustness.

MSVD-v2 dataset reduces errors and aids research.

Abstract

Natural Language Video Description (NLVD) has recently received strong interest in the Computer Vision, Natural Language Processing (NLP), Multimedia, and Autonomous Robotics communities. The State-of-the-Art (SotA) approaches obtained remarkable results when tested on the benchmark datasets. However, those approaches poorly generalize to new datasets. In addition, none of the existing works focus on the processing of the input to the NLVD systems, which is both visual and textual. In this work, it is presented an extensive study dealing with the role of the visual input, evaluated with respect to the overall NLP performance. This is achieved performing data augmentation of the visual component, applying common transformations to model camera distortions, noise, lighting, and camera positioning, that are typical in real-world operative scenarios. A t-SNE based analysis is proposed to evaluate the effects of the considered transformations on the overall visual data distribution. For this study, it is considered the English subset of Microsoft Research Video Description (MSVD) dataset, which is used commonly for NLVD. It was observed that this dataset contains a relevant amount of syntactic and semantic errors. These errors have been amended manually, and the new version of the dataset (called MSVD-v2) is used in the experimentation. The MSVD-v2 dataset is released to help to gain insight into the NLVD problem.