Logical Negation Augmenting and Debiasing for Prompt-based Methods

TL;DR

This paper identifies logical negation as a key challenge in prompt-based NLP reasoning and proposes NAND, a method that introduces negative propositions to improve logical reasoning without retraining.

Contribution

The paper introduces NAND, a simple technique that mitigates negation bias in prompt-based methods, enhancing logical reasoning capabilities without model parameter updates.

Findings

NAND effectively calibrates logical negation in prompt-based methods.

NAND significantly improves logical reasoning performance across datasets.

The method does not require retraining of models.

Abstract

Prompt-based methods have gained increasing attention on NLP and shown validity on many downstream tasks. Many works have focused on mining these methods' potential for knowledge extraction, but few explore their ability to make logical reasoning. In this work, we focus on the effectiveness of the prompt-based methods on first-order logical reasoning and find that the bottleneck lies in logical negation. Based on our analysis, logical negation tends to result in spurious correlations to negative answers, while propositions without logical negation correlate to positive answers. To solve the problem, we propose a simple but effective method, Negation Augmenting and Negation Debiasing (NAND), which introduces negative propositions to prompt-based methods without updating parameters. Specifically, these negative propositions can counteract spurious correlations by providing "not" for all instances so that models cannot make decisions only by whether expressions contain a logical negation. Experiments on three datasets show that NAND not only solves the problem of calibrating logical negation but also significantly enhances prompt-based methods of logical reasoning without model retraining.