URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates

TL;DR

This paper introduces the URL benchmark for evaluating pretrained models' ability to transfer both representations and uncertainty estimates across datasets, highlighting current strengths and challenges in transferable uncertainty quantification.

Contribution

It proposes a new benchmark for assessing transferable uncertainty estimates in pretrained models and evaluates multiple uncertainty quantifiers on this benchmark.

Findings

Approaches focusing on uncertainty of representations outperform probability-based methods.

Transferable uncertainty quantification remains an open challenge.

Transferability of uncertainty estimates does not conflict with traditional representation goals.

Abstract

Representation learning has significantly driven the field to develop pretrained models that can act as a valuable starting point when transferring to new datasets. With the rising demand for reliable machine learning and uncertainty quantification, there is a need for pretrained models that not only provide embeddings but also transferable uncertainty estimates. To guide the development of such models, we propose the Uncertainty-aware Representation Learning (URL) benchmark. Besides the transferability of the representations, it also measures the zero-shot transferability of the uncertainty estimate using a novel metric. We apply URL to evaluate eleven uncertainty quantifiers that are pretrained on ImageNet and transferred to eight downstream datasets. We find that approaches that focus on the uncertainty of the representation itself or estimate the prediction risk directly outperform those that are based on the probabilities of upstream classes. Yet, achieving transferable uncertainty quantification remains an open challenge. Our findings indicate that it is not necessarily in conflict with traditional representation learning goals. Code is provided under https://github.com/mkirchhof/url .