Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence

TL;DR

This paper investigates the logical negation property in large language models, finds they often violate it, and proposes a novel meaning-matching training task to improve their lexical semantic understanding and downstream task performance.

Contribution

The paper introduces a new meaning-matching training task that enhances language models' understanding of lexical semantics and logical negation, outperforming previous methods.

Findings

PLMs frequently violate the logical negation property.

The meaning-matching task improves lexical semantic learning.

Fine-tuning with the task maintains or improves downstream performance.

Abstract

The logical negation property (LNP), which implies generating different predictions for semantically opposite inputs, is an important property that a trustworthy language model must satisfy. However, much recent evidence shows that large-size pre-trained language models (PLMs) do not satisfy this property. In this paper, we perform experiments using probing tasks to assess PLM's LNP understanding. Unlike previous studies that only examined negation expressions, we expand the boundary of the investigation to lexical semantics. Through experiments, we observe that PLMs violate the LNP frequently. To alleviate the issue, we propose a novel intermediate training task, names meaning-matching, designed to directly learn a meaning-text correspondence, instead of relying on the distributional hypothesis. Through multiple experiments, we find that the task enables PLMs to learn lexical semantic information. Also, through fine-tuning experiments on 7 GLUE tasks, we confirm that it is a safe intermediate task that guarantees a similar or better performance of downstream tasks. Finally, we observe that our proposed approach outperforms our previous counterparts despite its time and resource efficiency.