Kandinsky Conformal Prediction: Efficient Calibration of Image Segmentation Algorithms

TL;DR

Kandinsky calibration leverages spatial image structure to efficiently calibrate image segmentation classifiers, especially useful when calibration data is scarce, improving coverage accuracy over traditional methods.

Contribution

The paper introduces Kandinsky calibration, a novel method that uses spatial correlations in images to enhance calibration efficiency for segmentation models with limited data.

Findings

Kandinsky calibration improves coverage accuracy in segmentation tasks.

It outperforms pixelwise and imagewise calibration with small datasets.

The method is effective on MS-COCO and Medical Decathlon datasets.

Abstract

Image segmentation algorithms can be understood as a collection of pixel classifiers, for which the outcomes of nearby pixels are correlated. Classifier models can be calibrated using Inductive Conformal Prediction, but this requires holding back a sufficiently large calibration dataset for computing the distribution of non-conformity scores of the model's predictions. If one only requires only marginal calibration on the image level, this calibration set consists of all individual pixels in the images available for calibration. However, if the goal is to attain proper calibration for each individual pixel classifier, the calibration set consists of individual images. In a scenario where data are scarce (such as the medical domain), it may not always be possible to set aside sufficiently many images for this pixel-level calibration. The method we propose, dubbed ``Kandinsky calibration'', makes use of the spatial structure present in the distribution of natural images to simultaneously calibrate the classifiers of ``similar'' pixels. This can be seen as an intermediate approach between marginal (imagewise) and conditional (pixelwise) calibration, where non-conformity scores are aggregated over similar image regions, thereby making more efficient use of the images available for calibration. We run experiments on segmentation algorithms trained and calibrated on subsets of the public MS-COCO and Medical Decathlon datasets, demonstrating that Kandinsky calibration method can significantly improve the coverage. When compared to both pixelwise and imagewise calibration on little data, the Kandinsky method achieves much lower coverage errors, indicating the data efficiency of the Kandinsky calibration.