Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization

TL;DR

This paper introduces Defeasible Visual Entailment (DVE), a new task that allows models to modify image-text entailment relationships based on updates, with novel evaluators and optimization methods improving accuracy and reliability.

Contribution

It presents the first exploration of defeasible reasoning in visual entailment, along with a new evaluator and reward-driven optimization for better model updates.

Findings

The proposed evaluator effectively captures entailment changes.

Reward-driven optimization improves update quality.

Experimental results show enhanced accuracy and reliability.

Abstract

We introduce a new task called Defeasible Visual Entailment (DVE), where the goal is to allow the modification of the entailment relationship between an image premise and a text hypothesis based on an additional update. While this concept is well-established in Natural Language Inference, it remains unexplored in visual entailment. At a high level, DVE enables models to refine their initial interpretations, leading to improved accuracy and reliability in various applications such as detecting misleading information in images, enhancing visual question answering, and refining decision-making processes in autonomous systems. Existing metrics do not adequately capture the change in the entailment relationship brought by updates. To address this, we propose a novel inference-aware evaluator designed to capture changes in entailment strength induced by updates, using pairwise contrastive learning and categorical information learning. Additionally, we introduce a reward-driven update optimization method to further enhance the quality of updates generated by multimodal models. Experimental results demonstrate the effectiveness of our proposed evaluator and optimization method.