Evaluate Confidence Instead of Perplexity for Zero-shot Commonsense Reasoning

TL;DR

This paper introduces Non-Replacement Confidence (NRC), a new metric for zero-shot commonsense reasoning that outperforms traditional perplexity-based methods by better capturing contextual integrity in language models.

Contribution

It proposes NRC, a novel confidence measure based on ELECTRA's RTD objective, improving zero-shot commonsense reasoning performance over perplexity metrics.

Findings

NRC significantly improves zero-shot reasoning accuracy.

RTD-based PLMs possess essential commonsense knowledge.

NRC outperforms perplexity on multiple benchmarks.

Abstract

Commonsense reasoning is an appealing topic in natural language processing (NLP) as it plays a fundamental role in supporting the human-like actions of NLP systems. With large-scale language models as the backbone, unsupervised pre-training on numerous corpora shows the potential to capture commonsense knowledge. Current pre-trained language model (PLM)-based reasoning follows the traditional practice using perplexity metric. However, commonsense reasoning is more than existing probability evaluation, which is biased by word frequency. This paper reconsiders the nature of commonsense reasoning and proposes a novel commonsense reasoning metric, Non-Replacement Confidence (NRC). In detail, it works on PLMs according to the Replaced Token Detection (RTD) pre-training objective in ELECTRA, in which the corruption detection objective reflects the confidence on contextual integrity that is more relevant to commonsense reasoning than existing probability. Our proposed novel method boosts zero-shot performance on two commonsense reasoning benchmark datasets and further seven commonsense question-answering datasets. Our analysis shows that pre-endowed commonsense knowledge, especially for RTD-based PLMs, is essential in downstream reasoning.