Resolving Word Vagueness with Scenario-guided Adapter for Natural Language Inference

TL;DR

This paper introduces ScenaFuse, a scenario-guided adapter that integrates visual information with linguistic knowledge to enhance natural language inference, addressing language vagueness and improving model understanding.

Contribution

The paper presents a novel ScenaFuse adapter that combines visual and linguistic data within pre-trained models for improved NLI performance.

Findings

ScenaFuse improves NLI accuracy on benchmark datasets.

The image-sentence interaction module enhances cross-modal understanding.

Adaptive fusion of visual and semantic info boosts inference capabilities.

Abstract

Natural Language Inference (NLI) is a crucial task in natural language processing that involves determining the relationship between two sentences, typically referred to as the premise and the hypothesis. However, traditional NLI models solely rely on the semantic information inherent in independent sentences and lack relevant situational visual information, which can hinder a complete understanding of the intended meaning of the sentences due to the ambiguity and vagueness of language. To address this challenge, we propose an innovative ScenaFuse adapter that simultaneously integrates large-scale pre-trained linguistic knowledge and relevant visual information for NLI tasks. Specifically, we first design an image-sentence interaction module to incorporate visuals into the attention mechanism of the pre-trained model, allowing the two modalities to interact comprehensively. Furthermore, we introduce an image-sentence fusion module that can adaptively integrate visual information from images and semantic information from sentences. By incorporating relevant visual information and leveraging linguistic knowledge, our approach bridges the gap between language and vision, leading to improved understanding and inference capabilities in NLI tasks. Extensive benchmark experiments demonstrate that our proposed ScenaFuse, a scenario-guided approach, consistently boosts NLI performance.