Crowdsourcing Universal Part-Of-Speech Tags for Code-Switching

TL;DR

This paper presents a crowdsourcing methodology for annotating part-of-speech tags in code-switching speech data, achieving high agreement and recall, which enhances NLP resources for bilingual communities.

Contribution

It introduces a novel crowdsourcing approach tailored for multilingual code-switching data, including a cascaded annotation process and quality control measures.

Findings

High annotation agreement (0.95-0.96) with gold labels.

Recall across POS tags ranges from 0.87 to 0.99.

Effective adaptation of annotation process for multilingual data.

Abstract

Code-switching is the phenomenon by which bilingual speakers switch between multiple languages during communication. The importance of developing language technologies for codeswitching data is immense, given the large populations that routinely code-switch. High-quality linguistic annotations are extremely valuable for any NLP task, and performance is often limited by the amount of high-quality labeled data. However, little such data exists for code-switching. In this paper, we describe crowd-sourcing universal part-of-speech tags for the Miami Bangor Corpus of Spanish-English code-switched speech. We split the annotation task into three subtasks: one in which a subset of tokens are labeled automatically, one in which questions are specifically designed to disambiguate a subset of high frequency words, and a more general cascaded approach for the remaining data in which questions are displayed to the worker following a decision tree structure. Each subtask is extended and adapted for a multilingual setting and the universal tagset. The quality of the annotation process is measured using hidden check questions annotated with gold labels. The overall agreement between gold standard labels and the majority vote is between 0.95 and 0.96 for just three labels and the average recall across part-of-speech tags is between 0.87 and 0.99, depending on the task.