Grounded Textual Entailment

TL;DR

This paper explores whether incorporating visual information into textual entailment models improves their understanding, using a multimodal SNLI dataset and comparing models with and without visual context.

Contribution

It introduces a visually-grounded version of the textual entailment task and demonstrates that visual data can enhance model performance, highlighting current limitations.

Findings

Visual information improves entailment model accuracy.

Current multimodal models do not optimally ground visual context.

Error analysis reveals areas for model improvement.

Abstract

Capturing semantic relations between sentences, such as entailment, is a long-standing challenge for computational semantics. Logic-based models analyse entailment in terms of possible worlds (interpretations, or situations) where a premise P entails a hypothesis H iff in all worlds where P is true, H is also true. Statistical models view this relationship probabilistically, addressing it in terms of whether a human would likely infer H from P. In this paper, we wish to bridge these two perspectives, by arguing for a visually-grounded version of the Textual Entailment task. Specifically, we ask whether models can perform better if, in addition to P and H, there is also an image (corresponding to the relevant "world" or "situation"). We use a multimodal version of the SNLI dataset (Bowman et al., 2015) and we compare "blind" and visually-augmented models of textual entailment. We show that visual information is beneficial, but we also conduct an in-depth error analysis that reveals that current multimodal models are not performing "grounding" in an optimal fashion.