Dealing with Uncertainties in User Feedback: Strategies Between Denying and Accepting

TL;DR

This paper examines strategies for managing unreliable user feedback in web systems, highlighting their limitations and proposing a new acceptance-based approach within a probabilistic framework to improve assessment accuracy.

Contribution

It critically evaluates existing strategies for handling human uncertainty and introduces a novel acceptance-based method to better address feedback unreliability.

Findings

Existing strategies are largely ineffective in handling human uncertainty.

A probabilistic framework and hypothesis testing reveal limitations of current methods.

A new acceptance strategy shows potential benefits over filtering approaches.

Abstract

Latest research revealed a considerable lack of reliability within user feedback and discussed striking impacts for the assessment of adaptive web systems and content personalisation approaches, e.g. ranking errors, systematic biases to accuracy metrics as well as its natural offset (the magic barrier). In order to perform holistic assessments and to improve web systems, a variety of strategies have been proposed to deal with this so-called human uncertainty. In this contribution we discuss the most relevant strategies to handle uncertain feedback and demonstrate that these approaches are more or less ineffective to fulfil their objectives. In doing so, we consider human uncertainty within a purely probabilistic framework and utilise hypothesis testing as well as a generalisation of the magic barrier to compare the effects of recently proposed algorithms. On this basis we recommend a novel strategy of acceptance which turns away from mere filtering and discuss potential benefits for the community of the WWW.