Ordinal Common-sense Inference

TL;DR

This paper introduces a new framework for evaluating and modeling common-sense inference in natural language, focusing on predicting the likelihood of inferences and constructing an ordinal entailment dataset using neural models.

Contribution

It proposes an ordinal inference framework, creates a new dataset for ordinal entailment, and trains neural models to predict and generate likely inferences based on common sense.

Findings

Neural models can effectively score and generate plausible inferences.

Ordinal annotation reveals nuanced differences in common-sense likelihood.

The dataset enables more fine-grained evaluation of inference models.

Abstract

Humans have the capacity to draw common-sense inferences from natural language: various things that are likely but not certain to hold based on established discourse, and are rarely stated explicitly. We propose an evaluation of automated common-sense inference based on an extension of recognizing textual entailment: predicting ordinal human responses on the subjective likelihood of an inference holding in a given context. We describe a framework for extracting common-sense knowledge from corpora, which is then used to construct a dataset for this ordinal entailment task. We train a neural sequence-to-sequence model on this dataset, which we use to score and generate possible inferences. Further, we annotate subsets of previously established datasets via our ordinal annotation protocol in order to then analyze the distinctions between these and what we have constructed.