Wisdom of the crowd from unsupervised dimension reduction

TL;DR

This paper demonstrates that crowd wisdom can be modeled as an unsupervised dimension reduction problem, enabling the use of techniques like PCA and Isomap to improve collective decision accuracy across various response types.

Contribution

It unifies crowd wisdom with unsupervised dimension reduction, introducing new, broadly applicable methods that outperform existing solutions, including supervised approaches.

Findings

Unsupervised dimension reduction techniques can effectively model crowd wisdom.

Methods like PCA and Isomap outperform traditional and supervised models.

The approach applies to both binary and continuous responses.

Abstract

Wisdom of the crowd, the collective intelligence derived from responses of multiple human or machine individuals to the same questions, can be more accurate than each individual, and improve social decision-making and prediction accuracy. This can also integrate multiple programs or datasets, each as an individual, for the same predictive questions. Crowd wisdom estimates each individual's independent error level arising from their limited knowledge, and finds the crowd consensus that minimizes the overall error. However, previous studies have merely built isolated, problem-specific models with limited generalizability, and mainly for binary (yes/no) responses. Here we show with simulation and real-world data that the crowd wisdom problem is analogous to one-dimensional unsupervised dimension reduction in machine learning. This provides a natural class of crowd wisdom solutions, such as principal component analysis and Isomap, which can handle binary and also continuous responses, like confidence levels, and consequently can be more accurate than existing solutions. They can even outperform supervised-learning-based collective intelligence that is calibrated on historical performance of individuals, e.g. penalized linear regression and random forest. This study unifies crowd wisdom and unsupervised dimension reduction, and thereupon introduces a broad range of highly-performing and widely-applicable crowd wisdom methods. As the costs for data acquisition and processing rapidly decrease, this study will promote and guide crowd wisdom applications in the social and natural sciences, including data fusion, meta-analysis, crowd-sourcing, and committee decision making.