Experiments on Crowdsourcing Policy Assessment

TL;DR

This study evaluates the effectiveness of non-expert crowds from virtual labor markets in assessing climate change policies, comparing their performance to experts and analyzing the impact of geographic relevance.

Contribution

It introduces an experimental framework for comparing non-expert crowds with expert assessments in policy evaluation, considering geographic relevance as a variable.

Findings

Non-expert crowds can approximate expert policy assessments.

Geographic relevance influences crowd assessment accuracy.

Crowd composition impacts evaluation outcomes.

Abstract

Can Crowds serve as useful allies in policy design? How do non-expert Crowds perform relative to experts in the assessment of policy measures? Does the geographic location of non-expert Crowds, with relevance to the policy context, alter the performance of non-experts Crowds in the assessment of policy measures? In this work, we investigate these questions by undertaking experiments designed to replicate expert policy assessments with non-expert Crowds recruited from Virtual Labor Markets. We use a set of ninety-six climate change adaptation policy measures previously evaluated by experts in the Netherlands as our control condition to conduct experiments using two discrete sets of non-expert Crowds recruited from Virtual Labor Markets. We vary the composition of our non-expert Crowds along two conditions: participants recruited from a geographical location directly relevant to the policy context and participants recruited at-large. We discuss our research methods in detail and provide the findings of our experiments.