Rethinking Crowd Sourcing for Semantic Similarity

TL;DR

This paper examines the ambiguities in crowd-sourced semantic similarity labeling, emphasizing the impact of binary annotator perceptions and proposing heuristics to improve label reliability in NLP tasks.

Contribution

It identifies the dominant role of binary annotators in semantic similarity labeling and introduces heuristics to filter unreliable annotators, enhancing label quality.

Findings

Binary annotators significantly influence crowd-sourced labels.

Heuristics can effectively filter unreliable annotators.

Discussion on human perception of semantic similarity.

Abstract

Estimation of semantic similarity is crucial for a variety of natural language processing (NLP) tasks. In the absence of a general theory of semantic information, many papers rely on human annotators as the source of ground truth for semantic similarity estimation. This paper investigates the ambiguities inherent in crowd-sourced semantic labeling. It shows that annotators that treat semantic similarity as a binary category (two sentences are either similar or not similar and there is no middle ground) play the most important role in the labeling. The paper offers heuristics to filter out unreliable annotators and stimulates further discussions on human perception of semantic similarity.