FC-CONAN: An Exhaustively Paired Dataset for Robust Evaluation of Retrieval Systems

TL;DR

FC-CONAN is a comprehensive dataset that exhaustively pairs hate speech messages with counter-narratives, enabling more accurate evaluation of counterspeech retrieval systems and revealing previously unlabelled positive pairs.

Contribution

It introduces the first exhaustively paired hate speech and counter-narrative dataset, improving evaluation fidelity and enabling detailed error analysis.

Findings

Uncovered hundreds of previously unlabelled positive pairs.

Created four annotation partitions balancing reliability and scale.

Facilitates more faithful evaluation of counterspeech retrieval systems.

Abstract

Hate speech (HS) is a critical issue in online discourse, and one promising strategy to counter it is through the use of counter-narratives (CNs). Datasets linking HS with CNs are essential for advancing counterspeech research. However, even flagship resources like CONAN (Chung et al., 2019) annotate only a sparse subset of all possible HS-CN pairs, limiting evaluation. We introduce FC-CONAN (Fully Connected CONAN), the first dataset created by exhaustively considering all combinations of 45 English HS messages and 129 CNs. A two-stage annotation process involving nine annotators and four validators produces four partitions-Diamond, Gold, Silver, and Bronze-that balance reliability and scale. None of the labeled pairs overlap with CONAN, uncovering hundreds of previously unlabelled positives. FC-CONAN enables more faithful evaluation of counterspeech retrieval systems and facilitates detailed error analysis. The dataset is publicly available.