Modeling Event Plausibility with Consistent Conceptual Abstraction

TL;DR

This paper investigates the inconsistency of Transformer-based models in judging event plausibility across conceptual classes and proposes a post-hoc method to improve their consistency and alignment with human judgments.

Contribution

It identifies the inconsistency issue in plausibility models across conceptual classes and introduces a simple post-hoc technique to enhance model consistency and human correlation.

Findings

Transformer models are inconsistent across conceptual classes.

Injecting lexical knowledge does not fully resolve inconsistency.

Post-hoc adjustment improves model plausibility correlation with humans.

Abstract

Understanding natural language requires common sense, one aspect of which is the ability to discern the plausibility of events. While distributional models -- most recently pre-trained, Transformer language models -- have demonstrated improvements in modeling event plausibility, their performance still falls short of humans'. In this work, we show that Transformer-based plausibility models are markedly inconsistent across the conceptual classes of a lexical hierarchy, inferring that "a person breathing" is plausible while "a dentist breathing" is not, for example. We find this inconsistency persists even when models are softly injected with lexical knowledge, and we present a simple post-hoc method of forcing model consistency that improves correlation with human plausibility judgements.