SURF: Improving classifiers in production by learning from busy and noisy end users

TL;DR

This paper introduces SURF, a new algorithm designed to improve classifier performance in production by effectively handling noisy and incomplete user feedback, addressing the challenge of user non-response.

Contribution

The paper presents SURF, a novel algorithm that manages user non-response in feedback data, enhancing classifier retraining in noisy, real-world settings.

Findings

SURF outperforms traditional crowdsourcing algorithms in non-response scenarios.

The algorithm effectively reduces noise from user feedback.

Experimental results show improved classifier accuracy after using SURF.

Abstract

Supervised learning classifiers inevitably make mistakes in production, perhaps mis-labeling an email, or flagging an otherwise routine transaction as fraudulent. It is vital that the end users of such a system are provided with a means of relabeling data points that they deem to have been mislabeled. The classifier can then be retrained on the relabeled data points in the hope of performance improvement. To reduce noise in this feedback data, well known algorithms from the crowdsourcing literature can be employed. However, the feedback setting provides a new challenge: how do we know what to do in the case of user non-response? If a user provides us with no feedback on a label then it can be dangerous to assume they implicitly agree: a user can be busy, lazy, or no longer a user of the system! We show that conventional crowdsourcing algorithms struggle in this user feedback setting, and present a new algorithm, SURF, that can cope with this non-response ambiguity.