# Assessment of Prediction Techniques: The Impact of Human Uncertainty

## TL;DR

This paper examines how human uncertainty affects the evaluation of predictive models in data mining, proposing an uncertainty-aware approach that shifts from point estimates to distribution-based assessments, with experiments in recommender systems.

## Contribution

It introduces a methodology that incorporates human uncertainty into model evaluation, improving the accuracy and interpretability of predictive performance assessments.

## Key findings

- Human uncertainty significantly impacts prediction quality metrics.
- Distribution-based evaluation provides more reliable assessments.
- Experiments demonstrate the benefits of the proposed approach.

## Abstract

Many data mining approaches aim at modelling and predicting human behaviour. An important quantity of interest is the quality of model-based predictions, e.g. for finding a competition winner with best prediction performance. In real life, human beings meet their decisions with considerable uncertainty. Its assessment and resulting implications for statistically evident evaluation of predictive models are in the main focus of this contribution. We identify relevant sources of uncertainty as well as the limited ability of its accurate measurement, propose an uncertainty-aware methodology for more evident evaluations of data mining approaches, and discuss its implications for existing quality assessment strategies. Specifically, our approach switches from common point-paradigm to more appropriate distribution-paradigm. This is exemplified in the context of recommender systems and their established metrics of prediction quality. The discussion is substantiated by comprehensive experiments with real users, large-scale simulations, and discussion of prior evaluation campaigns (i.a. Netflix Prize) in the light of human uncertainty aspects.