Machine Truth Serum

TL;DR

This paper introduces machine learning methods inspired by Bayesian Truth Serum to identify true answers in classification tasks, especially when the true answer is in the minority, improving over traditional majority voting ensembles.

Contribution

It proposes novel machine learning techniques that detect when minority answers are correct, enhancing ensemble accuracy beyond majority voting and existing algorithms.

Findings

Methods outperform traditional ensemble algorithms.

Better accuracy achieved when minority answers are true.

Applicable as a subroutine in ensemble methods.

Abstract

Wisdom of the crowd revealed a striking fact that the majority answer from a crowd is often more accurate than any individual expert. We observed the same story in machine learning--ensemble methods leverage this idea to combine multiple learning algorithms to obtain better classification performance. Among many popular examples is the celebrated Random Forest, which applies the majority voting rule in aggregating different decision trees to make the final prediction. Nonetheless, these aggregation rules would fail when the majority is more likely to be wrong. In this paper, we extend the idea proposed in Bayesian Truth Serum that "a surprisingly more popular answer is more likely the true answer" to classification problems. The challenge for us is to define or detect when an answer should be considered as being "surprising". We present two machine learning aided methods which aim to reveal the truth when it is minority instead of majority who has the true answer. Our experiments over real-world datasets show that better classification performance can be obtained compared to always trusting the majority voting. Our proposed methods also outperform popular ensemble algorithms. Our approach can be generically applied as a subroutine in ensemble methods to replace majority voting rule.