Deploying a Robust Active Preference Elicitation Algorithm on MTurk: Experiment Design, Interface, and Evaluation for COVID-19 Patient Prioritization

TL;DR

This study validates a robust preference elicitation algorithm with real users on MTurk, demonstrating its effectiveness in prioritizing COVID-19 patient policies over random query methods.

Contribution

It extends prior simulation-based work by empirically testing a preference elicitation method with real users in a critical healthcare context.

Findings

Robust method outperforms random queries by 21% in utility-based policy recommendations.

Experiment conducted with 193 MTurk workers.

Method shows practical effectiveness in real-world preference elicitation.

Abstract

Preference elicitation leverages AI or optimization to learn stakeholder preferences in settings ranging from marketing to public policy. The online robust preference elicitation procedure of arXiv:2003.01899 has been shown in simulation to outperform various other elicitation procedures in terms of effectively learning individuals' true utilities. However, as with any simulation, the method makes a series of assumptions that cannot easily be verified to hold true beyond simulation. Thus, we propose to validate the robust method's performance using real users, focusing on the particular challenge of selecting policies for prioritizing COVID-19 patients for scarce hospital resources during the pandemic. To this end, we develop an online platform for preference elicitation where users report their preferences between alternatives over a moderate number of pairwise comparisons chosen by a particular elicitation procedure. We recruit 193 Amazon Mechanical Turk (MTurk) workers to report their preferences and demonstrate that the robust method outperforms asking random queries by 21%, the next best performing method in the simulated results of arXiv:2003.01899, in terms of recommending policies with a higher utility.