Robustness Evaluation of Regression Tasks with Skewed Domain Preferences

TL;DR

This paper evaluates the robustness of regression models in skewed domains where extreme values are more relevant, addressing challenges in performance assessment under non-uniform preferences and distribution uncertainties.

Contribution

It introduces methods to assess regression model performance considering skewed preferences and distribution uncertainty, highlighting their impact on experimental conclusions.

Findings

Performance varies with relevance of target values.

Proposed methods improve robustness evaluation.

Extreme values significantly influence model assessment.

Abstract

In natural phenomena, data distributions often deviate from normality. One can think of cataclysms as a self-explanatory example: events that occur almost never, and at the same time are many standard deviations away from the common outcome. In many scientific contexts it is exactly these tail events that researchers are most interested in anticipating, so that adequate measures can be taken to prevent or attenuate a major impact on society. Despite such efforts, we have yet to provide definite answers to crucial issues in evaluating predictive solutions in domains such as weather, pollution, health. In this paper, we deal with two encapsulated problems simultaneously. First, assessing the performance of regression models when non-uniform preferences apply - not all values are equally relevant concerning the accuracy of their prediction, and there's a particular interest in the most extreme values. Second, assessing the robustness of models when dealing with uncertainty regarding the actual underlying distribution of values relevant for such problems. We show how different levels of relevance associated with target values may impact experimental conclusions, and demonstrate the practical utility of the proposed methods.