Lexicosyntactic Inference in Neural Models

TL;DR

This paper examines how neural models handle lexicosyntactic inferences, specifically in factuality prediction, revealing systematic errors and providing a new dataset for evaluation.

Contribution

It introduces a comprehensive factuality judgment dataset for English clause-embedding verbs and analyzes neural models' performance on this task.

Findings

Neural models exhibit systematic errors in factuality prediction.

The dataset covers all English clause-embedding verbs in various contexts.

Current models struggle with certain lexicosyntactic inferences.

Abstract

We investigate neural models' ability to capture lexicosyntactic inferences: inferences triggered by the interaction of lexical and syntactic information. We take the task of event factuality prediction as a case study and build a factuality judgment dataset for all English clause-embedding verbs in various syntactic contexts. We use this dataset, which we make publicly available, to probe the behavior of current state-of-the-art neural systems, showing that these systems make certain systematic errors that are clearly visible through the lens of factuality prediction.