CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm

TL;DR

This paper introduces CIKQA, a benchmark for evaluating models' ability to perform commonsense inference and knowledge sufficiency across diverse tasks using a unified QA format, emphasizing knowledge understanding and generalization.

Contribution

It proposes a novel benchmark that separates knowledge acquisition from inference, aligning tasks with knowledge bases and assessing models' inference and generalization capabilities.

Findings

Models can identify if knowledge is sufficient for tasks.

Unified QA format enables cross-task generalization evaluation.

Annotations reveal gaps in models' commonsense inference.

Abstract

Recently, the community has achieved substantial progress on many commonsense reasoning benchmarks. However, it is still unclear what is learned from the training process: the knowledge, inference capability, or both? We argue that due to the large scale of commonsense knowledge, it is infeasible to annotate a large enough training set for each task to cover all commonsense for learning. Thus we should separate the commonsense knowledge acquisition and inference over commonsense knowledge as two separate tasks. In this work, we focus on investigating models' commonsense inference capabilities from two perspectives: (1) Whether models can know if the knowledge they have is enough to solve the task; (2) Whether models can develop commonsense inference capabilities that generalize across commonsense tasks. We first align commonsense tasks with relevant knowledge from commonsense knowledge bases and ask humans to annotate whether the knowledge is enough or not. Then, we convert different commonsense tasks into a unified question answering format to evaluate models' generalization capabilities. We name the benchmark as Commonsense Inference with Knowledge-in-the-loop Question Answering (CIKQA).