Approximating Wisdom of Crowds using K-RBMs

TL;DR

This paper introduces K-RBMs, a novel clustering-based approach to aggregate noisy crowd-sourced labels, improving the estimation of true labels in tasks like web search quality and product rating.

Contribution

It presents a new method linking GMMs and RBMs for label aggregation, offering a probabilistic clustering approach to improve crowd label quality.

Findings

K-RBMs effectively cluster noisy labels

The method improves label accuracy over baseline approaches

Empirical results on real datasets validate the approach

Abstract

An important way to make large training sets is to gather noisy labels from crowds of non experts. We propose a method to aggregate noisy labels collected from a crowd of workers or annotators. Eliciting labels is important in tasks such as judging web search quality and rating products. Our method assumes that labels are generated by a probability distribution over items and labels. We formulate the method by drawing parallels between Gaussian Mixture Models (GMMs) and Restricted Boltzmann Machines (RBMs) and show that the problem of vote aggregation can be viewed as one of clustering. We use K-RBMs to perform clustering. We finally show some empirical evaluations over real datasets.