Continuous Entailment Patterns for Lexical Inference in Context

TL;DR

This paper introduces CONAN, a method using continuous patterns to improve lexical inference in context, outperforming discrete patterns and setting new state-of-the-art results on LIiC benchmarks.

Contribution

The paper proposes a novel continuous pattern approach (CONAN) that enhances PLM performance on lexical inference tasks by allowing flexible token representations outside the vocabulary.

Findings

CONAN outperforms discrete pattern methods on LIiC benchmarks.

Continuous patterns improve PLM performance in lexical inference.

Insights into pattern types that benefit PLMs in natural language understanding.

Abstract

Combining a pretrained language model (PLM) with textual patterns has been shown to help in both zero- and few-shot settings. For zero-shot performance, it makes sense to design patterns that closely resemble the text seen during self-supervised pretraining because the model has never seen anything else. Supervised training allows for more flexibility. If we allow for tokens outside the PLM's vocabulary, patterns can be adapted more flexibly to a PLM's idiosyncrasies. Contrasting patterns where a "token" can be any continuous vector vs. those where a discrete choice between vocabulary elements has to be made, we call our method CONtinuous pAtterNs (CONAN). We evaluate CONAN on two established benchmarks for lexical inference in context (LIiC) a.k.a. predicate entailment, a challenging natural language understanding task with relatively small training sets. In a direct comparison with discrete patterns, CONAN consistently leads to improved performance, setting a new state of the art. Our experiments give valuable insights into the kind of pattern that enhances a PLM's performance on LIiC and raise important questions regarding our understanding of PLMs using text patterns.