Distributed Conformal Prediction via Message Passing

TL;DR

This paper introduces two message-passing algorithms for distributed conformal prediction that enable reliable, calibrated inference across decentralized networks with limited data and communication constraints.

Contribution

It proposes novel distributed conformal prediction methods, Q-DCP and H-DCP, tailored for decentralized settings with communication constraints, enhancing calibration and coverage guarantees.

Findings

Q-DCP accelerates convergence with smoothing and regularization.

H-DCP achieves consensus-based histogram estimation.

Trade-offs identified between communication, coverage, and prediction set size.

Abstract

Post-hoc calibration of pre-trained models is critical for ensuring reliable inference, especially in safety-critical domains such as healthcare. Conformal Prediction (CP) offers a robust post-hoc calibration framework, providing distribution-free statistical coverage guarantees for prediction sets by leveraging held-out datasets. In this work, we address a decentralized setting where each device has limited calibration data and can communicate only with its neighbors over an arbitrary graph topology. We propose two message-passing-based approaches for achieving reliable inference via CP: quantile-based distributed conformal prediction (Q-DCP) and histogram-based distributed conformal prediction (H-DCP). Q-DCP employs distributed quantile regression enhanced with tailored smoothing and regularization terms to accelerate convergence, while H-DCP uses a consensus-based histogram estimation approach. Through extensive experiments, we investigate the trade-offs between hyperparameter tuning requirements, communication overhead, coverage guarantees, and prediction set sizes across different network topologies. The code of our work is released on: https://github.com/HaifengWen/Distributed-Conformal-Prediction.