How Nonconformity Functions and Difficulty of Datasets Impact the Efficiency of Conformal Classifiers

TL;DR

This paper investigates how nonconformity functions and dataset difficulty influence the efficiency of conformal classifiers, proposing a method to combine nonconformity functions and evaluating across multiple algorithms and datasets.

Contribution

It extends previous studies by experimentally analyzing various classifiers and datasets, and introduces a new method to combine nonconformity functions for improved efficiency.

Findings

Inverse probability and margin functions have different impacts on prediction singleton rates.

The relationship between nonconformity functions and efficiency varies with dataset and classifier.

A new combined nonconformity function improves conformal classifier performance.

Abstract

The property of conformal predictors to guarantee the required accuracy rate makes this framework attractive in various practical applications. However, this property is achieved at a price of reduction in precision. In the case of conformal classification, the systems can output multiple class labels instead of one. It is also known from the literature, that the choice of nonconformity function has a major impact on the efficiency of conformal classifiers. Recently, it was shown that different model-agnostic nonconformity functions result in conformal classifiers with different characteristics. For a Neural Network-based conformal classifier, the inverse probability (or hinge loss) allows minimizing the average number of predicted labels, and margin results in a larger fraction of singleton predictions. In this work, we aim to further extend this study. We perform an experimental evaluation using 8 different classification algorithms and discuss when the previously observed relationship holds or not. Additionally, we propose a successful method to combine the properties of these two nonconformity functions. The experimental evaluation is done using 11 real and 5 synthetic datasets.