SemEval-2020 Task 5: Counterfactual Recognition

TL;DR

This paper introduces a new benchmark task for recognizing and extracting counterfactual statements in natural language, with two subtasks focusing on identification and extraction of antecedents and consequents.

Contribution

It defines the first shared task on counterfactual recognition, providing datasets, baseline code, and evaluation metrics to advance research in this area.

Findings

27 submissions to Subtask-1

11 submissions to Subtask-2

Benchmark resources publicly available

Abstract

We present a counterfactual recognition (CR) task, the shared Task 5 of SemEval-2020. Counterfactuals describe potential outcomes (consequents) produced by actions or circumstances that did not happen or cannot happen and are counter to the facts (antecedent). Counterfactual thinking is an important characteristic of the human cognitive system; it connects antecedents and consequents with causal relations. Our task provides a benchmark for counterfactual recognition in natural language with two subtasks. Subtask-1 aims to determine whether a given sentence is a counterfactual statement or not. Subtask-2 requires the participating systems to extract the antecedent and consequent in a given counterfactual statement. During the SemEval-2020 official evaluation period, we received 27 submissions to Subtask-1 and 11 to Subtask-2. The data, baseline code, and leaderboard can be found at https://competitions.codalab.org/competitions/21691. The data and baseline code are also available at https://zenodo.org/record/3932442.