Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions

TL;DR

This paper reviews the state-of-the-art in natural language generation from visual events, emphasizing the importance of modeling interactions between visual sequences and language, and discusses open challenges and future research directions.

Contribution

It presents a unified perspective on various visual-language tasks, surveys recent approaches, and highlights key open questions in modeling visual events for language generation.

Findings

Multiple tasks are unified under the broader problem of modeling visual event-language interactions.

Current approaches face common challenges in understanding temporal and multimodal relationships.

The paper identifies open research questions and suggests future directions for the field.

Abstract

In recent years, a substantial body of work in visually grounded natural language processing has focused on real-life multimodal scenarios such as describing content depicted in images or videos. However, comparatively less attention has been devoted to study the nature and degree of interaction between the different modalities in these scenarios. In this paper, we argue that any task dealing with natural language generation from sequences of images or frames is an instance of the broader, more general problem of modeling the intricate relationships between visual events unfolding over time and the features of the language used to interpret, describe, or narrate them. Therefore, solving these tasks requires models to be capable of identifying and managing such intricacies. We consider five seemingly different tasks, which we argue are compelling instances of this broader multimodal problem. Subsequently, we survey the modeling and evaluation approaches adopted for these tasks in recent years and examine the common set of challenges these tasks pose. Building on this perspective, we identify key open questions and propose several research directions for future investigation.