A Comprehensive Review of the Video-to-Text Problem

TL;DR

This paper provides a comprehensive review of the video-to-text problem, covering state-of-the-art methods, datasets, evaluation metrics, challenges, and future directions in connecting visual videos with textual descriptions.

Contribution

It categorizes and analyzes existing techniques, evaluates 26 datasets, and discusses progress, challenges, and future research avenues in video-to-text tasks.

Findings

Progress in video captioning and retrieval techniques

Identification of dataset limitations and strengths

Discussion of key challenges and future directions

Abstract

Research in the Vision and Language area encompasses challenging topics that seek to connect visual and textual information. When the visual information is related to videos, this takes us into Video-Text Research, which includes several challenging tasks such as video question answering, video summarization with natural language, and video-to-text and text-to-video conversion. This paper reviews the video-to-text problem, in which the goal is to associate an input video with its textual description. This association can be mainly made by retrieving the most relevant descriptions from a corpus or generating a new one given a context video. These two ways represent essential tasks for Computer Vision and Natural Language Processing communities, called text retrieval from video task and video captioning/description task. These two tasks are substantially more complex than predicting or retrieving a single sentence from an image. The spatiotemporal information present in videos introduces diversity and complexity regarding the visual content and the structure of associated language descriptions. This review categorizes and describes the state-of-the-art techniques for the video-to-text problem. It covers the main video-to-text methods and the ways to evaluate their performance. We analyze twenty-six benchmark datasets, showing their drawbacks and strengths for the problem requirements. We also show the progress that researchers have made on each dataset, we cover the challenges in the field, and we discuss future research directions.