Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity Recognition

TL;DR

This paper treats crowdsourced annotations as domain-specific data, applying domain adaptation techniques to improve named entity recognition, achieving state-of-the-art results with minimal expert annotations.

Contribution

It introduces an annotator-aware domain adaptation approach for crowdsourced NER, bridging crowdsourcing and domain adaptation methods for improved learning.

Findings

Achieves state-of-the-art NER performance on a crowdsourced dataset.

Effective with minimal expert annotations under supervised setting.

Demonstrates the viability of domain adaptation techniques in crowdsourcing contexts.

Abstract

Crowdsourcing is regarded as one prospective solution for effective supervised learning, aiming to build large-scale annotated training data by crowd workers. Previous studies focus on reducing the influences from the noises of the crowdsourced annotations for supervised models. We take a different point in this work, regarding all crowdsourced annotations as gold-standard with respect to the individual annotators. In this way, we find that crowdsourcing could be highly similar to domain adaptation, and then the recent advances of cross-domain methods can be almost directly applied to crowdsourcing. Here we take named entity recognition (NER) as a study case, suggesting an annotator-aware representation learning model that inspired by the domain adaptation methods which attempt to capture effective domain-aware features. We investigate both unsupervised and supervised crowdsourcing learning, assuming that no or only small-scale expert annotations are available. Experimental results on a benchmark crowdsourced NER dataset show that our method is highly effective, leading to a new state-of-the-art performance. In addition, under the supervised setting, we can achieve impressive performance gains with only a very small scale of expert annotations.