# Does It Make Sense? And Why? A Pilot Study for Sense Making and Explanation

## TL;DR

This paper introduces a new benchmark for directly evaluating whether AI systems can distinguish sensible statements from nonsensical ones and identify reasons for their lack of sense, addressing a key challenge in natural language understanding.

## Contribution

It presents a novel benchmark for sense making and explanation in NLP, enabling direct assessment of systems' ability to differentiate and explain nonsensical statements.

## Key findings

- Models struggle to reliably identify nonsensical statements.
- Human performance exceeds that of current models.
- Different challenges exist for system sense making compared to humans.

## Abstract

Introducing common sense to natural language understanding systems has received increasing research attention. It remains a fundamental question on how to evaluate whether a system has a sense making capability. Existing benchmarks measures commonsense knowledge indirectly and without explanation. In this paper, we release a benchmark to directly test whether a system can differentiate natural language statements that make sense from those that do not make sense. In addition, a system is asked to identify the most crucial reason why a statement does not make sense. We evaluate models trained over large-scale language modeling tasks as well as human performance, showing that there are different challenges for system sense making.