# Natural Language Semantics With Pictures: Some Language & Vision Datasets and Potential Uses for Computational Semantics

## TL;DR

This paper explores how image-caption datasets can be repurposed to advance computational semantics by analyzing relations between expressions and images, enabling new learning and evaluation methods for grounded language understanding.

## Contribution

It introduces a novel perspective on existing image-caption corpora, demonstrating their potential for developing and assessing grounded semantic models using relational data.

## Key findings

- Linked image-expression relations can be used to learn grounded semantics.
- The 'linked to same image' relation captures semantic implications.
- An exemplar-model approach outperforms simple distributional models on derived datasets.

## Abstract

Propelling, and propelled by, the "deep learning revolution", recent years have seen the introduction of ever larger corpora of images annotated with natural language expressions. We survey some of these corpora, taking a perspective that reverses the usual directionality, as it were, by viewing the images as semantic annotation of the natural language expressions. We discuss datasets that can be derived from the corpora, and tasks of potential interest for computational semanticists that can be defined on those. In this, we make use of relations provided by the corpora (namely, the link between expression and image, and that between two expressions linked to the same image) and relations that we can add (similarity relations between expressions, or between images). Specifically, we show that in this way we can create data that can be used to learn and evaluate lexical and compositional grounded semantics, and we show that the "linked to same image" relation tracks a semantic implication relation that is recognisable to annotators even in the absence of the linking image as evidence. Finally, as an example of possible benefits of this approach, we show that an exemplar-model-based approach to implication beats a (simple) distributional space-based one on some derived datasets, while lending itself to explainability.