Identifying Chinese Opinion Expressions with Extremely-Noisy Crowdsourcing Annotations

TL;DR

This paper explores Chinese opinion expression identification using extremely-noisy crowdsourced annotations, proposing a novel mixup strategy to improve model robustness and demonstrating the effectiveness of crowdsourcing for this task.

Contribution

It introduces a synthetic training sample generation method via mixup to handle noisy crowdsourcing annotations in Chinese OEI.

Findings

Crowdsourcing is effective for Chinese opinion expression identification.

The proposed annotator-mixup improves model performance.

Synthetic data generation enhances training consistency.

Abstract

Recent works of opinion expression identification (OEI) rely heavily on the quality and scale of the manually-constructed training corpus, which could be extremely difficult to satisfy. Crowdsourcing is one practical solution for this problem, aiming to create a large-scale but quality-unguaranteed corpus. In this work, we investigate Chinese OEI with extremely-noisy crowdsourcing annotations, constructing a dataset at a very low cost. Following zhang et al. (2021), we train the annotator-adapter model by regarding all annotations as gold-standard in terms of crowd annotators, and test the model by using a synthetic expert, which is a mixture of all annotators. As this annotator-mixture for testing is never modeled explicitly in the training phase, we propose to generate synthetic training samples by a pertinent mixup strategy to make the training and testing highly consistent. The simulation experiments on our constructed dataset show that crowdsourcing is highly promising for OEI, and our proposed annotator-mixup can further enhance the crowdsourcing modeling.