TorchCP: A Python Library for Conformal Prediction

TL;DR

TorchCP is a comprehensive, GPU-accelerated Python library that integrates conformal prediction with deep learning models, enabling scalable uncertainty quantification for large-scale AI applications.

Contribution

It introduces TorchCP, a PyTorch-native library that supports advanced conformal prediction algorithms for deep neural networks, GNNs, and LLMs, with features like online prediction and GPU acceleration.

Findings

Achieves up to 90% reduction in inference time on large datasets

Supports a wide range of deep learning models with conformal prediction

Provides a scalable, GPU-compatible framework for uncertainty quantification

Abstract

Conformal prediction (CP) is a powerful statistical framework that generates prediction intervals or sets with guaranteed coverage probability. While CP algorithms have evolved beyond traditional classifiers and regressors to sophisticated deep learning models like deep neural networks (DNNs), graph neural networks (GNNs), and large language models (LLMs), existing CP libraries often lack the model support and scalability for large-scale deep learning (DL) scenarios. This paper introduces TorchCP, a PyTorch-native library designed to integrate state-of-the-art CP algorithms into DL techniques, including DNN-based classifiers/regressors, GNNs, and LLMs. Released under the LGPL-3.0 license, TorchCP comprises about 16k lines of code, validated with 100\% unit test coverage and detailed documentation. Notably, TorchCP enables CP-specific training algorithms, online prediction, and GPU-accelerated batch processing, achieving up to 90\% reduction in inference time on large datasets. With its low-coupling design, comprehensive suite of advanced methods, and full GPU scalability, TorchCP empowers researchers and practitioners to enhance uncertainty quantification across cutting-edge applications.