Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts

TL;DR

This study compares expert and non-expert judgments in ranking Swedish multi-word expressions, finding high correlation among all groups, indicating non-experts can reliably assess language difficulty.

Contribution

It demonstrates that non-experts' relative rankings of language expressions align closely with those of experts, challenging assumptions about expert-only assessments.

Findings

High correlation between expert and non-expert rankings

Non-experts' judgments are reliable for language difficulty assessment

Crowdsourcing can effectively gather linguistic difficulty data

Abstract

In this study we investigate to which degree experts and non-experts agree on questions of difficulty in a crowdsourcing experiment. We ask non-experts (second language learners of Swedish) and two groups of experts (teachers of Swedish as a second/foreign language and CEFR experts) to rank multi-word expressions in a crowdsourcing experiment. We find that the resulting rankings by all the three tested groups correlate to a very high degree, which suggests that judgments produced in a comparative setting are not influenced by professional insights into Swedish as a second language.