Crowdsourcing in Computer Vision

TL;DR

This survey reviews how crowdsourcing is used in computer vision to efficiently gather high-quality annotated data for various visual tasks, discussing methods, challenges, and future directions.

Contribution

It provides a comprehensive overview of crowdsourcing techniques, annotation quality assurance, and data collection strategies specific to computer vision research.

Findings

Crowdsourcing enables scalable data annotation for diverse vision tasks.

Effective interface design and data selection improve annotation quality and efficiency.

Future trends include advanced workflows and quality control methods.

Abstract

Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.