Knowledge Learning with Crowdsourcing: A Brief Review and Systematic Perspective

TL;DR

This paper systematically reviews the progress in crowdsourcing-based knowledge learning, highlighting technical advances across data, models, and processes, and providing future research directions for the AI community.

Contribution

It offers a comprehensive, systematic review of thirteen years of research on crowdsourcing learning, emphasizing key challenges, progress, and future blueprints.

Findings

Progress in data collection and quality control techniques

Development of models integrating crowdsourced information

Insights into learning process optimization

Abstract

Big data have the characteristics of enormous volume, high velocity, diversity, value-sparsity, and uncertainty, which lead the knowledge learning from them full of challenges. With the emergence of crowdsourcing, versatile information can be obtained on-demand so that the wisdom of crowds is easily involved to facilitate the knowledge learning process. During the past thirteen years, researchers in the AI community made great efforts to remove the obstacles in the field of learning from crowds. This concentrated survey paper comprehensively reviews the technical progress in crowdsourcing learning from a systematic perspective that includes three dimensions of data, models, and learning processes. In addition to reviewing existing important work, the paper places a particular emphasis on providing some promising blueprints on each dimension as well as discussing the lessons learned from our past research work, which will light up the way for new researchers and encourage them to pursue new contributions.