Robust Statistics meets elicitability: When fair model validation breaks down

TL;DR

This paper explores how robust statistics and elicitability interact, revealing that contamination in data can cause validation methods to fail, and proposes the need for validation procedures resilient to such issues.

Contribution

It introduces the elicitability breakdown point and analyzes why contaminated data undermines the objective validation of estimators, highlighting limitations of existing methods.

Findings

Elicitability often fails in contaminated data settings.

Robust trimming procedures can mitigate but not fully prevent validation breakdown.

Validation procedures with non-zero elicitability breakdown point are necessary.

Abstract

A crucial part of data analysis is the validation of the resulting estimators, in particular, if several competing estimators need to be compared. Whether an estimator can be objectively validated is not a trivial property. If there exists a loss function such that the theoretical risk is minimized by the quantity of interest, this quantity is called elicitable, allowing estimators for this quantity to be objectively validated and compared by evaluating such a loss function. Elicitability requires assumptions on the underlying distributions, often in the form of regularity conditions. Robust Statistics is a discipline that provides estimators in the presence of contaminated data. In this paper, we, introducing the elicitability breakdown point, formally pin down why the problems that contaminated data cause for estimation spill over to validation, letting elicitability fail. Furthermore, as the goal is usually to estimate the quantity of interest w.r.t. the non-contaminated distribution, even modified notions of elicitability may be doomed to fail. The performance of a trimming procedure that filters out instances from non-ideal distributions, which would be theoretically sound, is illustrated in several numerical experiments. Even in simple settings, elicitability however often fails, indicating the necessity to find validation procedures with non-zero elicitability breakdown point.