It's just a matter of perspective(s): Crowd-Powered Consensus Organization of Corpora

TL;DR

This paper presents ORCHESTRA, a cost-effective crowd-powered method for organizing collections of objects into consensus clusters despite diverse worker perspectives and partial views, outperforming existing algorithms.

Contribution

It introduces algorithms to identify consensus perspectives in crowd clustering and integrates them into an efficient workflow for organizing objects.

Findings

ORCHESTRA achieves better organization accuracy than existing methods.

It significantly reduces costs in crowd-based clustering tasks.

The approach effectively handles diverse worker perspectives and partial views.

Abstract

We study the problem of organizing a collection of objects - images, videos - into clusters, using crowdsourcing. This problem is notoriously hard for computers to do automatically, and even with crowd workers, is challenging to orchestrate: (a) workers may cluster based on different latent hierarchies or perspectives; (b) workers may cluster at different granularities even when clustering using the same perspective; and (c) workers may only see a small portion of the objects when deciding how to cluster them (and therefore have limited understanding of the "big picture"). We develop cost-efficient, accurate algorithms for identifying the consensus organization (i.e., the organizing perspective most workers prefer to employ), and incorporate these algorithms into a cost-effective workflow for organizing a collection of objects, termed ORCHESTRA. We compare our algorithms with other algorithms for clustering, on a variety of real-world datasets, and demonstrate that ORCHESTRA organizes items better and at significantly lower costs.