CrowDEA: Multi-view Idea Prioritization with Crowds

TL;DR

CrowDEA is a novel method that analyzes crowd-sourced idea evaluations to identify and visualize frontier ideas that excel in diverse latent criteria, accommodating evaluator disagreement and limited data.

Contribution

It introduces a multi-criteria embedding approach that estimates idea positions, evaluator preferences, and viewpoints to effectively prioritize and visualize diverse frontier ideas.

Findings

Effectively identifies frontier ideas across multiple criteria

Provides visualization of idea embeddings for better understanding

Handles limited evaluator data and diverse viewpoints

Abstract

Given a set of ideas collected from crowds with regard to an open-ended question, how can we organize and prioritize them in order to determine the preferred ones based on preference comparisons by crowd evaluators? As there are diverse latent criteria for the value of an idea, multiple ideas can be considered as "the best". In addition, evaluators can have different preference criteria, and their comparison results often disagree. In this paper, we propose an analysis method for obtaining a subset of ideas, which we call frontier ideas, that are the best in terms of at least one latent evaluation criterion. We propose an approach, called CrowDEA, which estimates the embeddings of the ideas in the multiple-criteria preference space, the best viewpoint for each idea, and preference criterion for each evaluator, to obtain a set of frontier ideas. Experimental results using real datasets containing numerous ideas or designs demonstrate that the proposed approach can effectively prioritize ideas from multiple viewpoints, thereby detecting frontier ideas. The embeddings of ideas learned by the proposed approach provide a visualization that facilitates observation of the frontier ideas. In addition, the proposed approach prioritizes ideas from a wider variety of viewpoints, whereas the baselines tend to use to the same viewpoints; it can also handle various viewpoints and prioritize ideas in situations where only a limited number of evaluators or labels are available.