A Survey on Uncertainty Toolkits for Deep Learning

TL;DR

This survey reviews 11 uncertainty estimation toolkits for deep learning, comparing their capabilities and highlighting the importance of standardized evaluation methods for trustworthy and reliable AI models.

Contribution

It is the first comprehensive survey focusing on uncertainty toolkits in deep learning, providing detailed comparisons and insights into their features and integration.

Findings

Pyro and TensorFlow Probability offer high flexibility and seamless integration.

Uncertainty Quantification 360 has broader methodological scope.

Toolkits can improve model trustworthiness and reproducibility.

Abstract

The success of deep learning (DL) fostered the creation of unifying frameworks such as tensorflow or pytorch as much as it was driven by their creation in return. Having common building blocks facilitates the exchange of, e.g., models or concepts and makes developments easier replicable. Nonetheless, robust and reliable evaluation and assessment of DL models has often proven challenging. This is at odds with their increasing safety relevance, which recently culminated in the field of "trustworthy ML". We believe that, among others, further unification of evaluation and safeguarding methodologies in terms of toolkits, i.e., small and specialized framework derivatives, might positively impact problems of trustworthiness as well as reproducibility. To this end, we present the first survey on toolkits for uncertainty estimation (UE) in DL, as UE forms a cornerstone in assessing model reliability. We investigate 11 toolkits with respect to modeling and evaluation capabilities, providing an in-depth comparison for the three most promising ones, namely Pyro, Tensorflow Probability, and Uncertainty Quantification 360. While the first two provide a large degree of flexibility and seamless integration into their respective framework, the last one has the larger methodological scope.