SPE: Symmetrical Prompt Enhancement for Fact Probing

TL;DR

This paper introduces Symmetrical Prompt Enhancement (SPE), a novel continuous prompt-based method that leverages task symmetry to improve factual probing of pretrained language models, demonstrating significant performance gains.

Contribution

The paper proposes SPE, a new method that constructs symmetrical prompts for subject and object prediction, enhancing factual knowledge probing in PLMs.

Findings

SPE outperforms previous probing methods on the LAMA dataset.

Symmetrical prompts improve the accuracy of factual probing.

The method demonstrates significant gains in factual knowledge retrieval.

Abstract

Pretrained language models (PLMs) have been shown to accumulate factual knowledge during pretrainingng (Petroni et al., 2019). Recent works probe PLMs for the extent of this knowledge through prompts either in discrete or continuous forms. However, these methods do not consider symmetry of the task: object prediction and subject prediction. In this work, we propose Symmetrical Prompt Enhancement (SPE), a continuous prompt-based method for factual probing in PLMs that leverages the symmetry of the task by constructing symmetrical prompts for subject and object prediction. Our results on a popular factual probing dataset, LAMA, show significant improvement of SPE over previous probing methods.