UncertaintyZoo: A Unified Toolkit for Quantifying Predictive Uncertainty in Deep Learning Systems

TL;DR

UncertaintyZoo is a comprehensive toolkit that integrates 29 uncertainty quantification methods for deep learning, enabling better assessment of prediction confidence across various models and applications.

Contribution

The paper introduces UncertaintyZoo, a unified platform that consolidates multiple UQ methods with a standardized interface, facilitating practical usage and research in uncertainty quantification.

Findings

Effectively reveals prediction uncertainty in code vulnerability detection.

Demonstrates usefulness of UQ methods on CodeBERT and ChatGLM3.

Provides a practical tool with a demonstration video.

Abstract

Large language models(LLMs) are increasingly expanding their real-world applications across domains, e.g., question answering, autonomous driving, and automatic software development. Despite this achievement, LLMs, as data-driven systems, often make incorrect predictions, which can lead to potential losses in safety-critical scenarios. To address this issue and measure the confidence of model outputs, multiple uncertainty quantification(UQ) criteria have been proposed. However, even though important, there are limited tools to integrate these methods, hindering the practical usage of UQ methods and future research in this domain. To bridge this gap, in this paper, we introduce UncertaintyZoo, a unified toolkit that integrates 29 uncertainty quantification methods, covering five major categories under a standardized interface. Using UncertaintyZoo, we evaluate the usefulness of existing uncertainty quantification methods under the code vulnerability detection task on CodeBERT and ChatGLM3 models. The results demonstrate that UncertaintyZoo effectively reveals prediction uncertainty. The tool with a demonstration video is available on the project site https://github.com/Paddingbuta/UncertaintyZoo.