Visually Grounded Language Learning: a review of language games, datasets, tasks, and models

TL;DR

This review systematically categorizes vision-language tasks into discriminative, generative, and interactive games, emphasizing the importance of interactive, embodied communication for grounding meanings in neural models.

Contribution

It introduces a novel categorization of vision-language tasks based on Wittgenstein's language games, highlighting future research directions in interactive and embodied models.

Findings

Interactive games are crucial for resolving ambiguities.

Physical embodiment enhances understanding of semantics.

Future work should focus on natural language communication in models.

Abstract

In recent years, several machine learning models have been proposed. They are trained with a language modelling objective on large-scale text-only data. With such pretraining, they can achieve impressive results on many Natural Language Understanding and Generation tasks. However, many facets of meaning cannot be learned by ``listening to the radio" only. In the literature, many Vision+Language (V+L) tasks have been defined with the aim of creating models that can ground symbols in the visual modality. In this work, we provide a systematic literature review of several tasks and models proposed in the V+L field. We rely on Wittgenstein's idea of `language games' to categorise such tasks into 3 different families: 1) discriminative games, 2) generative games, and 3) interactive games. Our analysis of the literature provides evidence that future work should be focusing on interactive games where communication in Natural Language is important to resolve ambiguities about object referents and action plans and that physical embodiment is essential to understand the semantics of situations and events. Overall, these represent key requirements for developing grounded meanings in neural models.