Evaluating Predictive Uncertainty and Robustness to Distributional Shift Using Real World Data

TL;DR

This paper emphasizes the importance of evaluating predictive uncertainty and robustness of machine learning models under distributional shifts, especially for regression tasks, and introduces new metrics and baseline evaluations using real-world weather data.

Contribution

It proposes novel metrics for assessing uncertainty and robustness in regression models under distributional shifts and evaluates baseline methods on a real-world weather dataset.

Findings

New metrics provide deeper insights into model performance under shifts

Baseline methods are evaluated, revealing strengths and weaknesses

Focus on regression tasks broadens applicability of robustness evaluation

Abstract

Most machine learning models operate under the assumption that the training, testing and deployment data is independent and identically distributed (i.i.d.). This assumption doesn't generally hold true in a natural setting. Usually, the deployment data is subject to various types of distributional shifts. The magnitude of a model's performance is proportional to this shift in the distribution of the dataset. Thus it becomes necessary to evaluate a model's uncertainty and robustness to distributional shifts to get a realistic estimate of its expected performance on real-world data. Present methods to evaluate uncertainty and model's robustness are lacking and often fail to paint the full picture. Moreover, most analysis so far has primarily focused on classification tasks. In this paper, we propose more insightful metrics for general regression tasks using the Shifts Weather Prediction Dataset. We also present an evaluation of the baseline methods using these metrics.