When Does Meaning Backfire? Investigating the Role of AMRs in NLI

TL;DR

This paper examines whether incorporating Abstract Meaning Representation (AMR) improves natural language inference (NLI) models, finding that AMR can hinder semantic understanding and instead amplify superficial differences, leading to potential misjudgments.

Contribution

It provides an empirical analysis of AMR's impact on NLI, revealing that AMR may not enhance semantic reasoning and can sometimes mislead models.

Findings

AMR integration in fine-tuning hampers model generalization.

Prompting with AMR yields slight improvements in GPT-4o.

AMR tends to amplify surface-level differences rather than semantic content.

Abstract

Natural Language Inference (NLI) relies heavily on adequately parsing the semantic content of the premise and hypothesis. In this work, we investigate whether adding semantic information in the form of an Abstract Meaning Representation (AMR) helps pretrained language models better generalize in NLI. Our experiments integrating AMR into NLI in both fine-tuning and prompting settings show that the presence of AMR in fine-tuning hinders model generalization while prompting with AMR leads to slight gains in GPT-4o. However, an ablation study reveals that the improvement comes from amplifying surface-level differences rather than aiding semantic reasoning. This amplification can mislead models to predict non-entailment even when the core meaning is preserved.