conformalClassification: A Conformal Prediction R Package for Classification

TL;DR

The conformalClassification R package provides tools for conformal prediction in classification, offering reliable confidence measures with options for transductive and inductive methods based on random forests.

Contribution

It introduces an R package implementing TCP and ICP for classification with random forests, including diagnostic tools and plans for extending to other algorithms.

Findings

TCP yields higher validity than ICP

The package effectively generates confidence measures for classification

Diagnostic tools help evaluate conformal prediction performance

Abstract

The conformalClassification package implements Transductive Conformal Prediction (TCP) and Inductive Conformal Prediction (ICP) for classification problems. Conformal Prediction (CP) is a framework that complements the predictions of machine learning algorithms with reliable measures of confidence. TCP gives results with higher validity than ICP, however ICP is computationally faster than TCP. The package conformalClassification is built upon the random forest method, where votes of the random forest for each class are considered as the conformity scores for each data point. Although the main aim of the conformalClassification package is to generate CP errors (p-values) for classification problems, the package also implements various diagnostic measures such as deviation from validity, error rate, efficiency, observed fuzziness and calibration plots. In future releases, we plan to extend the package to use other machine learning algorithms, (e.g. support vector machines) for model fitting.