AlignVE: Visual Entailment Recognition Based on Alignment Relations

TL;DR

AlignVE introduces an alignment matrix-based approach for visual entailment recognition, effectively modeling relation inference between images and hypotheses, and achieves state-of-the-art accuracy on SNLI-VE dataset.

Contribution

This paper presents a novel relation interaction architecture called AlignVE that explicitly models relation inference for visual entailment, improving over content-based methods.

Findings

Achieves 72.45% accuracy on SNLI-VE dataset.

Outperforms previous content-based models.

Demonstrates the effectiveness of relation alignment in VE tasks.

Abstract

Visual entailment (VE) is to recognize whether the semantics of a hypothesis text can be inferred from the given premise image, which is one special task among recent emerged vision and language understanding tasks. Currently, most of the existing VE approaches are derived from the methods of visual question answering. They recognize visual entailment by quantifying the similarity between the hypothesis and premise in the content semantic features from multi modalities. Such approaches, however, ignore the VE's unique nature of relation inference between the premise and hypothesis. Therefore, in this paper, a new architecture called AlignVE is proposed to solve the visual entailment problem with a relation interaction method. It models the relation between the premise and hypothesis as an alignment matrix. Then it introduces a pooling operation to get feature vectors with a fixed size. Finally, it goes through the fully-connected layer and normalization layer to complete the classification. Experiments show that our alignment-based architecture reaches 72.45\% accuracy on SNLI-VE dataset, outperforming previous content-based models under the same settings.