libconform v0.1.0: a Python library for conformal prediction

TL;DR

libconform v0.1.0 is a Python library that implements conformal prediction algorithms, providing an API and documentation, with future updates planned to enhance its stability and features.

Contribution

The paper introduces libconform v0.1.0, a new Python library for conformal prediction, detailing its algorithms, API, and implementation insights.

Findings

Provides conformal prediction algorithms in Python

Documents API and implementation details

Plans for future stability improvements

Abstract

This paper introduces libconform v0.1.0, a Python library for the conformal prediction framework, licensed under the MIT-license. libconform is not yet stable. This paper describes the main algorithms implemented and documents the API of libconform. Also some details about the implementation and changes in future versions are described.

Equations

$$\{\{\},\{0\},\{1\},\{0,1\}\}.$$

$$\Gamma :{\text{Z}}^{\ast }\times \text{X}\times (0,1)\to 2^{\text{Y}}.$$

$${\Gamma }^{{ϵ}_1}(\text{Lbag}{z}_1,\dots ,z_n\text{Rbag},x_{n+l})\subseteq {\Gamma }^{{ϵ}_2}(\text{Lbag}{z}_1,\dots ,z_n\text{Rbag},x_{n+l})$$

$$\frac{1}{o}\sum _{i=n+1}^{n+o}|{\Gamma }_i^{ϵ}|.$$

$${\alpha }_i=A_{n+1}(\text{Lbag}{z}_1,\dots ,z_{i-1},z_{i+1},\dots ,z_{n+1}\text{Rbag},z_i).$$

$${\alpha }_i=A_{n+1}(\text{Lbag}{z}_1,\dots ,z_{n+1}\text{Rbag},z_i).$$

$$\frac{|\{i=1,\dots ,n+1:{\alpha }_i\ge {\alpha }_{n+1}\}|}{n+1}.$$

$$\frac{|\{i=1,\dots ,n+1:{\alpha }_i>{\alpha }_{n+1}\}|+{\tau }_{n+1}|\{i=1,\dots ,n+1:{\alpha }_i={\alpha }_{n+1}\}|}{n+1}.$$

$$\displaystyle\alpha=A_{n}(\Lbag z_{1},\dots,z_{n}\Rbag,z)=\Delta\big{(}y,D_{\Lbag z_{1},\dots,z_{n}\Rbag}(x)\big{)},$$

$$\displaystyle\alpha_{i}=A_{n}(\Lbag z_{1},\dots,z_{i-1},z_{i+1},\dots,z_{n}\Rbag,z_{i})=\Delta\big{(}y_{i},D_{\Lbag z_{1},\dots,z_{i-1},z_{i+1},\dots,z_{n}\Rbag}(x_{i})\big{)}.$$

$$A_n(\text{Lbag}{z}_1,\dots ,z_n\text{Rbag},z)=\frac{{\text{min}}_{i=1,\dots ,n:y_i=y}d(x,x_i)}{{\text{min}}_{i=1,\dots ,n:y_i\ne y}d(x,x_i)},$$

$$A_n(\text{Lbag}{z}_1,\dots ,z_n\text{Rbag},z)=\frac{d_k^y}{d_k^{-y}},$$

$$S_i=\{\begin{array}{ll}[u_i,v_i]& \text{if }{b}_{n+1}>b_i\\ (-\infty ,u_i]\cup [v_i,\infty )& \text{if }{b}_{n+1}<b_i\\ [u_i,\infty )& \text{if }{b}_{n+1}=b_i>0\text{ and }{a}_{n+1}<a_i\\ (-\infty ,u_i]& \text{if }{b}_{n+1}=b_i>0\text{ and }{a}_{n+1}>a_i\\ \mathbb{R}& \text{if }{b}_{n+1}=b_i=0\text{ and }|a_{n+1}|\le |a_i|\\ \varnothing & \text{if }{b}_{n+1}=b_i=0\text{ and }|a_{n+1}|>|a_i|\end{array},$$

$$N_j^{\prime }=\{\begin{array}{ll}{N}_j-N_{j-1}& \text{if }j=0,\dots ,m\\ 0& \text{if }j=-1\end{array}$$

$$M_j^{\prime }=\{\begin{array}{ll}{M}_j-M_{j-1}& \text{if }j=1,\dots ,m\\ 0& \text{if }j=0\end{array},$$

$${\alpha }_i=\Delta (y_i,D_{\text{Lbag}{z}_1,\dots ,z_m\text{Rbag}}(x_i)),i=m+1,\dots ,n.$$

$$\frac{|\{i=m+1,\dots ,n:{\alpha }_i\ge {\alpha }_{n+l}\}|}{n-m+1},$$

$$\frac{|\{i=m+1,\dots ,n:{\alpha }_i>{\alpha }_{n+l}\}|+{\tau }_{n+l}|\{i=m+1,\dots ,n:{\alpha }_i={\alpha }_{n+l}\}|}{n-m+1}$$

$$\kappa :\mathbb{N}\times \text{Z}\to \text{K},$$

$$K_n:{\text{Z}}^n\to {\text{K}}^n.$$

$$\frac{|\{i:K_i=K_{n+1}\text{ \& }{\alpha }_i\ge {\alpha }_{n+1}\}|}{|\{i:K_i=K_{n+1}\}|}.$$

$$\frac{|\{i:K_i=K_{n+1}\text{ \& }{\alpha }_i>{\alpha }_{n+1}\}|+{\tau }_{n+1}|\{i:K_i=K_{n+1}\text{ \& }{\alpha }_i={\alpha }_{n+1}\}|}{|\{i:K_i=K_{n+1}\}|}.$$

$$A:{\text{K}}^{\ast }\times {\text{Z}}^{\ast }\times \text{K}\times \text{Z}\to \mathbb{R},$$

$$P_y=\frac{|\{(x_i,y_i)\in K:y_i=y\}|}{|K|}.$$

$$A=\frac{TP+TN}{TP+TN+FP+FN};P=\frac{TP}{TP+FP},$$

$$TPr=\frac{TP}{TP+FN},FPr=\frac{FP}{FP+TN}.$$

$$R=\frac{RP+RN}{RP+RU+TP+TN+FP+FN}.$$

Full text

Keywords: conformal prediction, Venn prediction, Python, reliable machine learning

1 Introduction

This paper introduces the Python library libconform, implementing concepts defined in Vovk et al. (2005), namely the conformal prediction framework and Venn prediction for reliable machine learning and predicting with certainty. These algorithms address a weakness of more traditional machine learning algorithms which produce only bare predictions, without their confidence in them/the probability of the prediction, therefore providing no measure of likelihood, desirable and even necessary in many real-world application domains.

The conformal prediction framework is composed of variations of the conformal prediction algorithm, first described in Vovk et al. (1999); Saunders et al. (1999). A conformal predictor provides a measurement of confidence in its predictions. A Venn predictor, on the other hand, provides a multi-probabilistic measurement, making it a multi-probabilistic predictor. Below in the text, Venn predictors are included if only “conformal prediction framework” is written, except stated otherwise.

The conformal prediction framework is applied successfully in many real-world domains, for example face recognition, medical diagnostic and prognostic and network traffic classification (see Balasubramanian et al., 2014, Part 3).

It is build on traditional machine learning algorithms, the so called underlying algorithms (see Papadopoulos et al., 2007), which makes Python the first choice for implementation, since its machine learning libraries are top of the class, still evolving and improving due to the commitment of a great community of developers and researchers.

libconform’s aim is to provide an easy to use, but very extensible API for the conformal prediction framework, so developers can use their preferred implementations for the underlying algorithm and can leverage the library, even in this early stage. libconform v0.1.0 is not yet stable; there are still features missing and the API is very likely to change and improve. The library is licensed under the MIT-license and its source code can be downloaded from \urlhttps://github.com/jofas/conform.

This paper combines libconform’s documentation with an outline of the implemented algorithms. At the end of each chapter there are notes on the implementation containing general information about the library, descriptions of the internal workings and the API and possible changes in future versions.

Appendix A provides an overview over libconform’s API and Appendix B contains examples on how to use the library.

2 Conformal predictors

Like stated in the introduction, this chapter will only outline conformal prediction (CP). For more details see Vovk et al. (2005).

Definition

CP—like the name suggests—determines the label(s) of an incoming observation based on how well it/they conform(s) with previous observed examples.

Let $\Lbag z_{1},\dots,z_{n}\Rbag$ be a bag, also called multiset111It is typical in machine learning to denote this as a data set, even though examples do not have to be unique, making the so called set a multiset. A multiset is not a list, since the ordering of the elements is not important. If training set, test set, etc. is written in this paper it always denotes a bag, not a set., of examples, where each example $z_{i}\in\textbf{Z}$ is a tuple $(x_{i},y_{i});x_{i}\in\textbf{X},y_{i}\in\textbf{Y}$. X is called the observation space and Y the label space. For this time Y is considered finite, making the task of prediction a classification task, rather than regression, which will be considered in Chapter 2.2.

Let $2^{\textbf{Y}}$ be the set of all subsets of Y, including the empty set. For example, let $\textbf{Y}:=\{0,1\}$; $2^{\textbf{Y}}$ would be:

[TABLE]

A conformal predictor can be defined as a confidence predictor $\Gamma$. A confidence predictor is a function

[TABLE]

$\textbf{Z}^{*}$ denotes a bag of examples with arbitrary length.

$\Gamma$ takes a bag of examples, a new observation which should be predicted and $\epsilon\in(0,1)$, the significance level, as its input and returns the so called prediction set. $1-\epsilon$ is called the confidence level (see Vovk et al., 2005, Chapter 2).

CP produces nested prediction sets. The prediction sets are called nested, because, for $\epsilon_{1}\geq\epsilon_{2}$, the prediction set of $\Gamma^{\epsilon_{1}}$ is a subset of $\Gamma^{\epsilon_{2}}$:

[TABLE]

(see Vovk et al., 2005, Chapter 2).

Online vs. offline setting

Conformal prediction can be used either in the online or the offline—or batch—learning setting.

In the online setting, after $\Gamma^{\epsilon}(\Lbag z_{1},\dots,z_{n}\Rbag,x_{n+1})$ has given its prediction, reality would return the true label $y^{\prime}$ for $x_{n+1}$. $z_{n+1}:=(x_{n+1},y^{\prime})$ would be appended to the bag before making the next prediction for $x_{n+2}$. In the offline setting $x_{n+2}$ would be computed without $z_{n+1}$ as part of the bag.

The offline learning setting, compared to online learning, weakens the validity and efficiency—outlined below in this chapter—of the predictor in favor of computational efficiency. Also, in many cases the pure online setting is not even possible or desired, since a predictor giving a prediction which is then validated from reality directly afterwards makes the predictor redundant (see Vovk et al., 2005, Chapter 4).

This topic is again discussed below in Chapter 3.

Transductive vs. inductive predictors

CP was designed to be transductive rather than inductive (see Vovk et al., 2005, Chapter 1).

An inductive predictor $D$ uses a training set $\Lbag z_{1},\dots,z_{n}\Rbag$ to deduce a decision surface or prediction rule it uses for predicting a new observation.

On the other hand a transductive predictor does no such thing, it rather uses all the previous seen examples from the training set to predict without deducing a decision surface beforehand.

While the transductive setting is more elegant than the inductive setting, it is computationally very expensive and not feasible for larger bags of examples and for use with underlying inductive algorithms—discussed in Chapter 2.1—which have a computationally complex training phase (see Papadopoulos et al., 2007; Vovk et al., 2005, Chapter 1).

Validity

A conformal predictor $\Gamma^{\epsilon}(\Lbag z_{1},\dots,z_{n}\Rbag,x_{n+1})$ in the online setting is conservatively valid under the exchangeability assumption. That means, as long as exchangeability holds, it makes errors at a frequency of $\epsilon$ or less. For more on exchangeability and the proof that CP is valid under exchangeability, refer to Vovk et al. (2005, Chapters 1–4, 7).

Efficiency

The efficiency of a conformal predictor can be determined with many criteria, for example the $N$ criterion. Let $\Lbag z_{n+1},\dots,z_{n+o}\Rbag$ be a test set. The $N$ criterion is the average size of the prediction sets for the test set:

[TABLE]

A small $N$ criteria is preferable (see Vovk et al., 2016).

Another criterion would be to determine the frequency of prediction sets with $|\Gamma^{\epsilon}_{i}|=1$. This is an important measure if only predictions with a single label are desired.

For more efficiency criteria see Vovk et al. (2016).

Nonconformity measures

In order to predict the label of a new observation $x_{n+1}$, $\Gamma^{\epsilon}(\Lbag z_{1},\dots,z_{n}\Rbag,x_{n+1})$ sets $z_{n+1}:=(x_{n+1},y)$, for each $y\in\textbf{Y}$ and checks how $z_{n+1}$ conforms with the examples of the bag $\Lbag z_{1},\dots,z_{n}\Rbag$.

This is done with a nonconformity measure $A_{n+1}:\textbf{Z}^{n}\times\textbf{Z}\rightarrow\mathbb{R}$. First, $z_{n+1}$ is added to the bag, then $A_{n+1}$ assigns a numerical score to each example in $z_{i}$:

[TABLE]

One can see in this equation that $z_{i}$ is removed from the bag. It is also possible to compute $\alpha_{i}$ with $z_{i}$ in the bag, which means for $A_{n+1}:\textbf{Z}^{n+1}\times\textbf{Z}\rightarrow\mathbb{R}$ the score is computed as:

[TABLE]

Which one is preferable is case-dependent (see Shafer and Vovk, 2008, Chapter 4.2.2).

$\alpha_{i}$ is called a nonconformity score.

p-values

The nonconformity scores can now be used to compute the p-value for $z_{n+1}$, which is the fraction of examples from the bag which are at least as nonconforming as $z_{n+1}$:

[TABLE]

If the fraction is close to the upper bound 1 the example $z_{n+1}$ is very conforming. On the other hand, if it is close to its lower bound $\frac{1}{n+1}$ it is quite nonconforming (see Vovk et al., 2005, Chapter 2).

Another way to determine the p-value is through smoothing, in which case the nonconformity scores equal to $\alpha_{n+1}$ are multiplied by a random value $\tau_{n+1}\in[0,1]$:

[TABLE]

A conformal predictor using the smoothed p-value is called a smoothed conformal predictor and is exactly valid under exchangeability in the online setting, which means it makes errors at a rate exactly $\epsilon$ (see Vovk et al., 2005, Chapter 2).

If the p-value of $z_{n+1}$ is larger than $\epsilon$, $y$ is added to the prediction set.

Notes on the implementation

libconform provides the CP class for creating conformal prediction classifiers. libconform’s classifier classes provide quite equal APIs, only with minor variations. The API of the predictor classes is comparable to major machine learning libraries like sklearn or keras (see Buitinck et al., 2013; Chollet et al., 2015).

It is common in machine learning to split the learning task in two distinct operations, first a training—or fit—operation on a bag of examples and then a predict operation on new observations. libconform’s predictor classes follow this style, providing a train and a predict method.

While this split in training and predicting is common for inductive classifiers, which first derive a prediction rule, or decision surface, from the training set and then predict unseen examples inductively based on that rule, it is not really the way CP works. CP was designed to be transductive, not inductive.

libconform’s aim is to be—one day—ready for production, where, for some application domains, the time complexity of predicting a new observation is crucial, while the time complexity of the training phase is—in a certain range—not as important. Therefore libconform’s CP class tries to minimize the time complexity of its predict method. Instead of adding $z_{n+1}$ to the bag and then computing $\alpha_{i}$ for each example in the bag during prediction, it computes $\alpha_{1},\dots,\alpha_{n}$ during training and only computes $\alpha_{n+1}$ in predict (see Algorithm 1, lines 3–5). Therefore—not adding $z_{n+1}$ to the bag—it currently computes the nonconformity scores based on $A_{n}$ instead of $A_{n+1}$.

Arguably CP does not implement the conformal prediction algorithm in its original form (transductive and online). It provides rather a offline conformal prediction implementation, or a special case of inductive conformal prediction, where the calibration set is equal to the whole bag of examples previously witnessed, instead of a subset (see Chapter 3). It is possible that CP will change to being the implementation of the original conformal prediction algorithm in a future version, or simply providing an extra method for the computationally more demanding original online learning setting (see Vovk et al., 2005, Chapter 2).

CP takes an instance of a nonconformity measure $A$ and a sequence of $\epsilon_{1},\dots,\epsilon_{g}$ as its arguments during initialization, therefore being the implementation of $\Gamma^{\epsilon_{1}},\dots,\Gamma^{\epsilon_{g}}$.

It also provides two extra utility methods for validation, score and score_online, which generate metrics for the conformal predictors $\Gamma^{\epsilon_{1}},\dots,\Gamma^{\epsilon_{g}}$. The most important of those metrics are the error rates $Err_{1},\dots,Err_{g}$. It the error rate $Err_{i}\leq\epsilon_{i}$ over the bag of examples provided to score/score_online than $\Gamma^{\epsilon_{i}}$ was valid on the bag.

score_online adds an example, after it was predicted, to the training set and calls train, using $\Gamma^{\epsilon_{1}},\dots,\Gamma^{\epsilon_{g}}$ in the online learning setting.

CP also provides a method for another setting of conformal prediction, this one not based on a significance level $\epsilon$: predict_best. predict_best always returns a single label, the one with the highest p-value and optionally also its significance level. The significance level is the second highest p-value, since a label is added to the prediction set—in the original setting—if its p-value is greater than $\epsilon$ (see Papadopoulos et al., 2007).

2.1 Nonconformity measures based on underlying

algorithms

Previously nonconformity measures were only described as any function $A:\textbf{Z}^{*}\times\textbf{Z}\rightarrow\mathbb{R}$, $\textbf{Z}^{*}$ being any possible bag of examples from Z. This chapter will make a more concrete description on what nonconformity measures are and how they use underlying traditional machine learning algorithms.

Let $D:\textbf{Z}^{*}\times\textbf{X}\rightarrow\hat{\textbf{Y}}$ be a traditional machine learning algorithm. $\hat{\textbf{Y}}$ must not be equal to Y. Furthermore there exists a discrepancy measure $\Delta:\textbf{Y}\times\hat{\textbf{Y}}\rightarrow\mathbb{R}$. For a concrete bag $\Lbag z_{1},\dots,z_{n}\Rbag$, $D_{\Lbag z_{1},\dots,z_{n}\Rbag}$ would be the instance of $D$ trained on the bag, generating a decision surface based on it. $D_{\Lbag z_{1},\dots,z_{n}\Rbag}(x)$ would return the label $\hat{y}$ for $x$. Now we can define the nonconformity score $\alpha$ for $z:=(x,y)$ from the nonconformity measure $A_{n}$ as:

[TABLE]

or rather with removed example for any $\alpha_{i}$, $i=1,\dots,n$:

[TABLE]

Especially the second equation can be computationally very complex since it requires to refit $D$ for each $i=1,\dots,n$. In general it is not very natural to use an inductive decision surface $D$ within the transductive framework of CP.

A popular nonconformity measure is based on the nearest neighbor method (see Vovk et al., 2005; Shafer and Vovk, 2008; Balasubramanian et al., 2014; Smirnov et al., 2009). The general description for the $k$-nearest neighbor method can be found in Smirnov et al. (2009), the other articles/books describe the nonconformity measure based on the 1-nearest neighbor method for $z:=(x,y)$ as:

[TABLE]

$d$ being a distance measure, for example the Euclidean distance. It should be noted that $A_{n}$ based on the 1-nearest neighbor method for $A_{n}(\Lbag z_{1},\dots,z_{n}\Rbag,z_{i}),i=1,\dots,n$ requires the removal of $z_{i}$ from the bag, since otherwise the smallest distance for $y_{i}=y_{j},j=1,\dots,n$ would always be 0 resulting in worthless nonconformity scores.

The more general nonconformity measure based on the $k$-nearest neighbor method can be described as:

[TABLE]

$d_{k}$ being the sum of the $k$ smallest distances to $x$, $-y$ being all the examples where $y\neq y_{i},i=1,\dots,n$.

Notes on the implementation

libconform tries again to be as extensible as possible, providing a way for developers to define their own nonconformity measures. For nonconformity measures libconform provides a module ncs containing predefined nonconformity measures and a base class for inheritance called NCSBase.

Predefined are currently the $k$-nearest neighbor method, one based on a decision tree (see Vovk et al., 2005, Chapter 4) and one based on neural networks (see Papadopoulos et al., 2007; Vovk et al., 2005, Chapter 4). The $k$-nearest neighbor method and the decision tree are based on the sklearn library (see Buitinck et al., 2013).

Nonconformity measures are classes inheriting from NCSBase and have to provide an interface with three methods: train, scores and score.

train$:\textbf{X}^{n}\times\textbf{Y}^{n}$ is for fitting the underlying algorithm $D$ to a bag of examples.

scores$:\textbf{X}^{m}\times\textbf{Y}^{m}\times bool\rightarrow\mathbb{R}^{m}$ returning the scores for a bag of examples. The $bool$ value provided as a parameter tells the nonconformity measure if the bag is equal to the bag provided to train, making it possible to implement (1), rather than (2). The CP-implementation passes the same bag to train and scores, while the inductive conformal prediction implementation (see Chapter 3) passes another bag—the so called calibration set—as a parameter to scores.

score$:\textbf{X}\times\textbf{Y}^{|\textbf{Y}|}\rightarrow\mathbb{R}^{|\textbf{Y}|}$ is for returning the scores of an example $x$ and each $y\in\textbf{Y}$ combined as $z:=(x,y)$.

2.2 Conformal predictor for regression: ridge

regression confidence machine

The ridge regression confidence machine algorithm is described in Nouretdinov et al. (2001, Chapter 2.3); Vovk et al. (2005, Chapter 2.3). In this chapter, unlike in the previous and following chapters, Y will be $\mathbb{R}$, making the prediction a regression problem instead of classification.

Algorithm 1 is not feasible for regression, since Y is now infinite and we would need to test for each $y\in\textbf{Y}$ if it is in the prediction set or not. Instead the ridge regression confidence machine (RRCM) algorithm offers a different approach, returning prediction intervals instead of prediction sets.

Even though RRCM has ridge regression in its name, it can be used with other underlying algorithms, like nearest neighbor regression. For more on ridge regression and its special case linear regression refer to e.g. Hastie et al. (2009, Chapter 3).

Let $\Lbag z_{1},\dots,z_{n}\Rbag$ be our bag of examples, let $z_{n+1}:=(x_{n+1},y)$ be the observation we want to predict and let $D_{\Lbag z_{1},\dots,z_{n+1}\Rbag}$ be an underlying regression algorithm. Previously nonconformity scores were treated as constants, now we treat them as functions, since $y$ is now an unknown variable: $\alpha_{i}(y)=|a_{i}+b_{i}y|,i=1,\dots,n+1$. $a_{i}$ and $b_{i}$ are provided by the underlying regression algorithm. Each $b_{i}$ is always positive, if not $a_{i}$ and $b_{i}$ are multiplied with $-1$.

Now we can compute the set of $y$’s which p-values are exceeding a significance level $\epsilon$. Let $S_{i}=\{y:|a_{i}+b_{i}y|\geq|a_{n+1}+b_{n+1}y|\},i=1,\dots,n$. Each $S_{i}$ looks like:

[TABLE]

so each $S_{i}$ is either an interval, a point (a special interval), the union of two rays, a ray, the real line or empty. $u_{i}$ and $v_{i}$ are either the minimum/maximum of $-\frac{a_{i}-a_{n+1}}{b_{i}-b_{n+1}}$ and $-\frac{a_{i}+a_{n+1}}{b_{i}+b_{n+1}}$, if $b_{n+1}\neq b_{i}$ or $u_{i}=v_{i}=-\frac{a_{i}+a_{n}}{2b_{i}}$, if $b_{n+1}=b_{i}>0$. The p-value can only change at $u_{i}$ or $v_{i}$. Therefore all $u_{i}$ and $v_{i}$ are sorted in ascending order generating the sequence $s_{1},\dots,s_{m}$ plus two more $s$-values, $s_{0}=-\infty$, $s_{m+1}=\infty$. The p-value is constant on any interval $(s_{i},s_{i+1}),i=0,\dots,m$ from the sorted set (see Nouretdinov et al., 2001).

After that $N$ and $M$ are computed from the sequence. $N_{j},j=0,\dots,m$ for the interval $(s_{j},s_{j+1})$ is the count of $S_{i}:(s_{j},s_{j+1})\subseteq S_{i},i=1,\dots,n$. $M_{j},j=1,\dots,m$, on the other hand, does the same count only for single $s_{j}$: $S_{i}:s_{j}\in S_{i},i=1,\dots,n$.

For a given significance level $\epsilon$ the prediction interval is the union of intervals from $N$ and points from $M$ for which $\frac{N_{j}}{n+1}>\epsilon$ or $\frac{M_{j}}{n+1}>\epsilon$, respectively (see Vovk et al., 2005, Chapter 2.3).

In Nouretdinov et al. (2001) it is stated that there could be holes in the prediction interval, which means the the RRCM would return more than a single prediction interval. According to the authors these holes rarely show in empirical tests. The authors therefore remark that the RRCM can just remove those holes—therefore returning a single interval—by simply returning the convex hull of the prediction intervals.

Notes on the implementation

The ridge regression confidence machine is implemented as a class RRCM. It provides the same API as CP. It implements a computationally less complex prediction method than the one described above. While the RRCM described above runs at $\mathcal{O}(n^{2})$, RRCM takes only $\mathcal{O}(n\log n)$, because it does not compute $N$ and $M$ directly but instead

[TABLE]

and

[TABLE]

which takes only $\mathcal{O}(n)$, making sorting the $u_{i}$ and $v_{i}$ values the most complex task (see Vovk et al., 2005, Chapter 2.3).

The RRCM implementation takes a flag during its initialization for dealing with the holes described above, so developers can choose if they want possibly more than one prediction interval or the convex hull.

RRCM is based on underlying regression algorithms providing it with its $a_{i}$ and $b_{i}$. Currently the library provides one of these regression algorithms, based on the $k$-nearest neighbor method. $a_{i}$ is the difference between $y_{i}$ and the average of the labels of its $k$-nearest neighbors. $y_{n+1}$ is set to 0, therefore the $k$-nearest neighbor method returns the negated average of the labels of the $k$-nearest neighbors of $x_{n+1}$ as $a_{n+1}$. For $b_{i},i=1,\dots,n$ it returns 0, for $b_{n+1}$ it returns 1.

For developing underlying regression scorers there exists a base class for inheritance called NCSBaseRegressor. It provides a comparable API to NCSBase—the base class for nonconformity measures—described in the previous chapter. Like NCSBase the API contains a train method for training the underlying algorithm. Instead of scores and score it has coeffs and coeffs_n. The first returns for a bag two vectors of coefficients $a_{i}$ and $b_{i}$ for each element in the bag. coeffs_n returns the coefficients for the observation that needs to be predicted, in this chapter $z_{n+1}:=(x_{n+1},y)$.

3 Inductive conformal predictors

Suppose we have an underlying inductive machine learning algorithm $D$ as our nonconformity measure and a bag $\Lbag z_{1},\dots,z_{n}\Rbag$ of examples. If we want to use $D$ as the nonconformity measure we need to fit it to our bag: $D_{\Lbag z_{1},\dots,z_{n}\Rbag}$. For some $D$ this can be a quite time consuming task and in general is not a very aesthetic thing to do in our transductive setting from the previous chapter, since—if we would want to predict a new observation $x_{n+1}$—we would need to refit $D$ for each $y\in\textbf{Y}$, because we compute our nonconformity scores adding $z_{n+1}:=(x_{n+1},y)$ to the bag and refitting $D$ with it (see Algorithm 1, lines 2–5). Even worse, if we would use (1) instead of (2) we would need to refit $D$ for each bag $\Lbag z_{1},\dots,z_{i-1},z_{i+1},\dots,z_{n+1}\Rbag,i=1,\dots,n$.

There exists a natural derivation from the transductive setting of conformal prediction to the inductive setting called inductive conformal prediction (ICP). ICP works more natural with nonconformity measures relying on inductive machine learning algorithms as the underlying algorithm (see Vovk et al., 2005, Chapter 4).

ICP is computationally less complex than CP, to the cost of the classifier’s validity and efficiency (see Vovk et al., 2005, Chapter 4).

Suppose, again, we have our bag of examples $\Lbag z_{1},\dots,z_{n}\Rbag$. ICP now splits this bag at a point $m<n$ into two bags, the so called training set $\Lbag z_{1},\dots,z_{m}\Rbag$ and the calibration set $\Lbag z_{m+1},\dots,z_{n}\Rbag$.

With the training set the underlying algorithm is trained generating $D_{\Lbag z_{1},\dots,z_{m}\Rbag}$. For each example in the calibration set the nonconformity score $\alpha_{i}$ gets computed:

[TABLE]

Now, for an incoming example $x_{n+l}$ set $z_{n+l}:=(x_{n+l},y)$ for each $y\in\textbf{Y}$ and compute the nonconformity score $\alpha_{n+l}$ like (5). The p-value of $z_{n+l}$ is

[TABLE]

or the smoothed version

[TABLE]

(see Papadopoulos et al., 2007).

The huge costs of fitting $D$ repetitively are now reduced to fitting $D$ only once. More elaborate update cycles—called teaching schedules—where $m$ is changing after certain events and how they impact the validity of the classifier can be found in Vovk et al. (2005, Chapters 4.3, 4.4).

Notes on the implementation

ICP is the class implementing inductive conformal prediction. It provides the same API as CP, except score_online. It has an additional method calibrate for generating the nonconformity scores for a the calibration set.

It works with the same nonconformity measures (instances of classes inheriting from NCSBase) as does CP.

Currently the nonconformity scores from the calibration set are saved internally as a vector. In future releases this will change to an optimized data structure for searching, e.g. a red-black tree (see Cormen et al., 2009).

4 Mondrian or conditional (inductive) conformal

predictors

The property of validity under the exchangeablity assumption can be further optimized with Mondrian or conditional (inductive) conformal prediction (MCP). In Vovk et al. (2005, Chapter 4.5) this form of conformal prediction is called Mondrian, in Balasubramanian et al. (2014, Chapter 2) it is called conditional, the only difference being the underlying taxonomy, which will be discussed below.

An example from Vovk et al. (2005, Chapter 4.5) makes it clear why the stronger form of validity provided by MCP can be important for some real-world application domains. The authors tested a 1-nearest neighbor based smoothed conformal predictor with the significance level $\epsilon=0.05$ on the USPS data set. The USPS data set contains 9298 images of handwritten digits. The observations are a $16\times 16$ matrix where each cell is in the interval of $(-1,1)$ and the labels obviously are 0 to 9 (see LeCun et al., 1989).

The authors found out, that while overall the validity held (the error frequency was nearly equal to $\epsilon=0.05$), the smoothed conformal predictor had an error rate of $11.7\%$ on examples with the label “5”. The smoothed conformal predictor masked its bad performance on examples with the label “5” simply by predicting other labels with an error rate of less than $\epsilon=0.05$, e.g. for the label “0” the error rate was below $0.01$ (see Vovk et al., 2005, Chapter 4.5).

The idea of MCP is to partition the examples into a discrete and finite set K of categories $k\in\textbf{K}$ and to achieve conditional validity in each category. For the partitioning a measurable function called a taxonomy is used. In Vovk et al. (2005, Chapter 4.5) the taxonomy is called Mondrian taxonomy and is defined as:

[TABLE]

in Balasubramanian et al. (2014, Chapter 2) the taxonomy is called a $n$-taxonomy:

[TABLE]

The Mondrian taxonomy $\kappa$ takes the index $i$ of an example $z_{i}$ from a sequence $z_{1},\dots,z_{n}$ and $z_{i}$ as its input and maps it to a category while the $n$-taxonomy $K_{n}$ takes a sequence of examples with a size of $n$ and maps it to a sequence of categories with size $n$. $K_{n}$ is more flexible than $\kappa$ since it is possible to make the decision which category an example from the sequence should be in based on the other examples from the sequence.

The $K$-conditional p-value for an example $z_{n+1}$ and a bag $\Lbag z_{1},\dots,z_{n}\Rbag$ is now defined for $i=1,\dots,n+1$ as:

[TABLE]

The smoothed version would be:

[TABLE]

(6) and (7) are the same for $\kappa$ if $K$ is substituted with $\kappa$.

A MCP classifier is category-wise valid under the exchangeability assumption (see Vovk et al., 2005; Balasubramanian et al., 2014).

Notes on the implementation

There is no direct implementation for MCP, libconform rather leverages the fact that CP and ICP are just a special form of Mondrian (inductive) conformal prediction, where $|\textbf{K}|=1$, which means all examples are in the same category. CP and ICP can take an argument during initialization called mondrian_taxonomy. Currently mondrian_taxonomy is a function—or a Python callable rather—which takes one example as its input and returns the category, basically a 1-taxonomy $K_{1}$ where the single example can only be looked at without context.

In practice a single example often is more information than really needed. Often just the label of the example is important, making the MCP based on this K a label conditional (inductive) conformal predictor (see Balasubramanian et al., 2014, Chapter 2).

mondrian_taxonomoy will change in a future version to $K_{n}$ for more flexibility.

Vovk et al. (2005, Chapter 4.5) defines Mondrian nonconformity measures

[TABLE]

which add the category to each example in order to compute the nonconformity scores. Currently libconform does not have an API for Mondrian nonconformity measures, which could change in future releases.

5 Multi-probabilistic prediction: Venn predictors

In the previous chapters we measured the likelihood of a prediction based on p-values. Even though they produce valid confidence predictions, the use of p-values is controversial and they have disadvantages compared to the probability of a prediction, namely that they are harder to reason about and that they are often confused with probabilities (see Vovk et al., 2005, Chapter 6.3).

The main negative property of probabilistic prediction is the fact that it is impossible to estimate true probabilities—under the unconstrained randomness assumption—from a finite bag of examples, if the objects of the bag do not precisely repeat themselves (see Vovk et al., 2005, Chapter 5).

To bypass this property and to achieve a notion of validity, Venn predictors produce a set of probability distributions $\{P_{y}|y\in\textbf{Y}\}$, $|\textbf{Y}|<\infty$ as their predictions, for which reason they are called multi-probabilistic predictors (see Balasubramanian et al., 2014, Chapter 2.8).

There are two definitions of validity for a Venn predictor, the stronger form of validity being that Venn predictors are “well-calibrated” (see Vovk et al., 2005, Chapter 6), while the weaker form of validity states that—under the unconstrained randomness assumption—a Venn predictor’s prediction is guaranteed to contain the conditional probability in its multi-probabilistic prediction with regard to the true probability distribution generating the examples (see Balasubramanian et al., 2014, Chapter 2.8).

A Venn predictor is based on a Venn taxonomy $V_{n}$ equal to the $n$-taxonomy described in the previous chapter. Now, suppose we want to predict the probability distribution for $z_{n+1}:=(x_{n+1},y);y\in\textbf{Y}$. We first determine the category $K\in\textbf{K}$ of $z_{n+1}$ with $V_{n+1}$ and then look at the frequency of $y$ in this particular category to generate the probability distribution:

[TABLE]

$K$ is not empty since it at least contains $z_{n+1}$. This is done for each $y\in\textbf{Y}$ generating the set of probability distributions $\{P_{y}|y\in\textbf{Y}\}$ which is returned as the multi-probability prediction. The label with the highest probability is the predicted label.

An example for such a Venn predictor is described in Vovk et al. (2005, Chapter 6.3). It is based on a Venn taxonomy using the 1-nearest neighbor method to map an example to a category. In this case the Venn taxonomy returns the label of the nearest neighbor as the category.

The Venn predictor generates a matrix $M:|\textbf{Y}|\times|\textbf{Y}|$. Each example $z_{n+1}:=(x_{n+1},y);y\in\textbf{Y}$ is mapped to a row. Each column contains the frequency of $y_{i}\in\textbf{Y}$ of all examples in the same category as $z_{n+1}$.

The quality of a column is its minimum entry. Now select the best column $M_{best}$ (with the highest quality) an return the label of the cell with the highest frequency as the label prediction and the column as the multi-probability prediction $\{P_{y}|y\in\textbf{Y}\}$ (see Vovk et al., 2005, Chapter 6.3).

Actually, in the example in Vovk et al. (2005, Chapter 6.3), instead of returning $\{P_{y}|y\in\textbf{Y}\}$ the Venn predictor returns the interval of the convex hull of the multi-probability prediction: $[min\ M_{best},\ max\ M_{best}]$. This is called the probability interval. The complementary interval $[1-max\ M_{best},\ 1-min\ M_{best}]$ is called the error probability interval.

Notes on the implementation

Venn prediction—like described above—is implemented as Venn. It again implements the same API as CP does. The only difference is that Venn’s predict method takes a flag proba. If proba is off only the label prediction is returned. On the other hand, if proba is set, the label prediction and the error probability interval is returned.

Currently Venn taxonomies have their own module vtx. A Venn taxonomy is an instance of a class that inherits from VTXBase, the same design like NCSBase and NCSBaseRegressor (see Chapter 2). Once the MCP implementation from the previous chapter moves from $1$-taxonomies $K_{1}$ to $n$-taxonomies $K_{n}$, Venn taxonomies and $n$-taxonomies will be combined—since they are equal—and libconform will provide a new API for both together.

6 Meta-conformal predictors

For understanding how meta-conformal predictors achieve reliability we have to introduce the concept of abstention and abstaining classifiers. An abstaining classifier uses a measure of uncertainty and if the uncertainty of a new observation is too high the classifier does not give a prediction, it rejects it (see Vanderlooy et al., 2009).

CP is easily modified to produce abstaining classifiers. We could simply predict the label with the highest p-value and if its significance level (the second highest p-value, see Chapter 2) is above a certain threshold, the classifier returns the label, otherwise it rejects the observation.

Meta-conformal predictors are described in Smirnov et al. (2009). The authors argue, that if there exists a classifier $B$ we would need to construct a nonconformity measure based on $B$; not an easy task, since there exists no approach for doing so in general.

Therefore they introduce the method of using a conformal predictor with an already established nonconformity measure $M$ as a meta-classifier in combination with $B$, so we can add a certainty measure to a prediction otherwise without any likelihood indicator. B:M is called the combined classifier.

Meta-classifier instances

The meta-classifier is a binary classifier that predicts new observations based on the meta data of the base classifier $B$. The labels of the meta data are either 0—the negative meta class—if $B$’s prediction for an observation $x_{i}$ was wrong or 1—the positive meta class—if it was correct.

Our meta-conformal predictor $M$ should use a certainty measure in order to decide for a new observation, if the prediction of the base classifier $B$ is trustworthy enough to return. Since $M$ is trained on our meta data with the positive and negative meta class, $M$ generates two p-values—one for each class—$p_{p}$, the positive p-value and $p_{n}$, the negative p-value. We can use both p-values to convert B:M to a scoring classifier with a score ratio $\frac{p_{p}}{p_{n}}$. Now we only need to define the reliability threshold $T$. If the score ratio generated by $M$ for a new observation is greater than $T$ we say the prediction of $B$ is trustworthy and is returned, otherwise B:M abstains from making the prediction (see Smirnov et al., 2009).

Performance metrics for binary classifiers

Metrics for the performance of a binary classifier can be given by a confusion matrix (see Figure 1). For a test set the confusion matrix counts the predicted examples and maps them—depending on the true class and the predicted class—to its entries.

From the confusion matrix we can derive interesting metrics for the binary classifier, the most important being the accuracy $A$, the precision rate $P$:

[TABLE]

the true positive rate $TPr$ and the false positive rate $FPr$:

[TABLE]

For a binary abstention classifier we can also add the rejection rate

[TABLE]

Constructing the combined classifier

B:M

In order to construct a combined classifier B:M we first need the meta data. For that we use the $k$-fold method normally used as a cross-validation technique (see Hastie et al., 2009, Chapter 7.10).

Let $\Lbag z_{1},\dots,z_{n}\Rbag,\ z_{i}\in\textbf{Z}:\textbf{X}\times\textbf{Y}$ be our training set. We split the training set in $k$ roughly equal sized partitions. For each partition: take the partition as test set, combine the others to a training set and fit $B$ to it. Let $B$ predict on the test set which then generates a partition of the meta data $\Lbag z^{\prime}_{i},\dots,z^{\prime}_{i+l}\Rbag,\ z^{\prime}_{i}\in\textbf{Z}^{\prime}:\textbf{X}\times\{0,1\}$ (see Algorithm 2).

After we are through with Algorithm 2 and we have generated our meta data $\Lbag z^{\prime}_{1},\dots,z^{\prime}_{n}\Rbag$, we potentially could train $M$ to it, but we still have to define the reliability threshold $T$.

Having a target accuracy $A_{t}$ Smirnov et al. (2009) proposes to use a ROC isometrics approach defined in Vanderlooy et al. (2009) to set $T$ based on $A_{t}$. In the case where we have a combined classifier B:M $A_{t}$ equals the precision rate $P_{M}$ of $M$ (see Smirnov et al., 2009).

Defining $T$ based on $P_{M}$ is done in five steps:

use the same $k$-fold algorithm used to generate the meta data—with $M$ instead of $B$—to generate a set of scoring ratios $\frac{p_{p}}{p_{n}}$. 2. 2.

construct the ROC curve based on the $TPr$ and $FPr$ of $M$ which indirectly maps to the scoring ratios. For more information about ROC curves see e.g. Fawcett (2006). 3. 3.

abstract the convex hull ROCCH of the ROC curve. 4. 4.

construct the iso-precision line from the equation $TPr=\frac{P_{M}}{1-P_{M}}\frac{N}{P}FPr$, $N$ being the number of negative meta instances, $P$ being the number of positive meta instances. This line represents all classifiers with a target precision rate of $P_{M}$. The classifier on the line which is abstaining the least is determined in the next step. 5. 5.

set $T$ as the scoring ratio $\frac{p_{p}}{p_{n}}$ at the intersection of the ROCCH and the iso-precision line.

After we have determined $T$, $B$ is trained on the whole training set $\Lbag z_{1},\dots,z_{n}\Rbag$, $M$ on the whole meta data set $\Lbag z^{\prime}_{1},\dots,z^{\prime}_{n}\Rbag$ and the training operation of B:M is over.

Notes on the implementation

libconform provides the Meta class for combined classifiers. In order to offer the most flexibility for developers, Meta only takes two interfaces to each classifier $B$ and $M$, one being a function used for training the classifier, the other being for predicting. This way Meta does not take the classifiers itself, so it can be used with any other library implementing $B$. The interfaces to $M$ could change in future versions, so Meta uses $M$ directly. Currently this is not the case since using conformal prediction and inductive conformal prediction takes different approaches.

The ROCCH is constructed using the scipy library’s implementation (based on the qhull library) of the Quickhull algorithm (see Jones et al., 2001–2019; Barber et al., 1996).

7 Conclusion

Like stated in the introduction, libconform v0.1.0 is not yet stable. The API is likely to change and improve and the library needs more tests. Furthermore there are some algorithms from the conformal prediction framework still missing, including:

• •

cross-conformal prediction (see Vovk, 2012)

• •

aggregated conformal prediction (see Carlsson et al., 2014)

• •

inductive ridge regression confidence machine (see Papadopoulos et al., 2002)

• •

Venn abers prediction (see Vovk and Petej, 2014)

• •

more nonconformity scores which come out of the box with libconform

Also the performance and especially multi-threading is an issue which will be dealt with in future versions.

All that said, currently libconform provides a very extensible API and already implements some of the main algorithms of the conformal prediction framework.

The goal for libconform is to be one day the go-to implementation of the conformal prediction framework for Python. Hopefully it can attract newcomers to use conformal prediction for their needs and will become a community project, which grows and improves constantly.

Appendices

Appendix A API reference

ncs

Module for nonconformity measures, both for classification and regression. Instances of nonconformity measures are needed by CP, ICP and RRCM in order for them to be able to predict.

Members

• •

ncs.base

Module containing the base classes from which nonconformity measurement implementations inherit.

Members

• –

ncs.base.NCSBase

Base class for nonconformity measures for classification. If a Python object is passed to CP or ICP as a nonconformity measure and the object does not inherit from NCSBase an exception is raised.

Methods

train(X, y)

Dummy method which needs to be implemented by the inheriting nonconformity measure.

This method is for training the underlying algorithm $D$ based on which the nonconformity measure generates the nonconformity scores.

[TABLE]

• *

scores(X, y, cp)

Dummy method which needs to be implemented by the inheriting nonconformity measure.

Computes the nonconformity scores for each example from a bag.

[TABLE]

• *

score(x, labels)

Dummy method which needs to be implemented by the inheriting nonconformity measure. Could change in a future version from the observation x to a bag of observations X.

This method computes the nonconformity score for each new example $z_{g}:=(x_{g},y),y\in\textbf{Y}$.

[TABLE]

• –

ncs.base.NCSBaseRegressor

Base class for nonconformity measures for regression. If a Python object is passed to RRCM as a nonconformity measure and the object does not inherit from NCSBaseRegressor an exception is raised.

Methods

train(X, y)

Dummy method which needs to be implemented by the inheriting nonconformity measure.

This method is for training the underlying algorithm $D$ based on which the nonconformity measure generates the nonconformity scores.

[TABLE]

• *

coeffs(X, y, cp)

Dummy method which needs to be implemented by the inheriting nonconformity measure.

Computes $a$ and $b$ for each example from a bag.

[TABLE]

• *

coeffs_n(x)

Dummy method which needs to be implemented by the inheriting nonconformity measure. Could change in a future version from the observation x to a bag of observations X.

Computes $a$ and $b$ for a new observation which should be predicted.

[TABLE]

• •

ncs.NCSDecisionTree(**sklearn)

Class implementing a nonconformity measure based on a decision tree for classification. The score for an example $z:=(x,y)$ is the amout of examples with the same label $y$ in the tree node containing $z$ devided through all the examples the tree node contains (see Vovk et al., 2005, Chapter 4).

The implementation is based on the scikit-learn implementation (see Buitinck et al., 2013).

[TABLE]

Methods

See \hyperref[itm:ncsbase] ncs.base.NCSBase.

• •

ncs.NCSKNearestNeighbors(**sklearn)

Class implementing a nonconformity measure based on the $k$-nearest neighbors method for classification. The score for an example $z:=(x,y)$ is computed as $\frac{d^{y}_{k}}{d^{-y}_{k}}$ (see Chapter 2.1).

The implementation is based on the scikit-learn implementation (see Buitinck et al., 2013).

[TABLE]

Methods

See \hyperref[itm:ncsbase] ncs.base.NCSBase.

• •

ncs.NCSKNearestNeighborsRegressor(**sklearn)

Class implementing a nonconformity measure based on the $k$-nearest neighbors method for regression (see Chapter 2.2).

The implementation is based on the scikit-learn implementation (see Buitinck et al., 2013).

[TABLE]

Methods

See \hyperref[itm:ncsbaseregressor] ncs.base.NCSBaseRegressor.

• •

ncs.NCSNeuralNet(train_, predict_, scorer = "sum", gamma = 0.0)

Class implementing a nonconformity measure based on a neural net for classification. The nonconformity score is computed via the output neurons and how their outputs defer. For more on nonconformity scores based on neural nets see Papadopoulos et al. (2007, Chapter 4); Vovk et al. (2005, Chapter 4).

[TABLE]

Methods

See \hyperref[itm:ncsbase] ncs.base.NCSBase.

vtx

Module for Venn taxonomies. Will be deprecated in future versions in favor of a module for $n$-taxonomies (see Chapters 4, 5).

Members

• •

vtx.base

Module containing the base class from which Venn taxonomies implementations inherit.

Members

• –

vtx.base.VTXBase

Base class for Venn taxonomies. If a Python object is passed to Venn as a Venn taxonmy and the object does not inherit from VTXBase an exception is raised.

Methods

train(X, y)

Dummy method which needs to be implemented by the inheriting Venn taxonomy.

This method is for training the underlying algorithm $D$ based on which a Venn taxonomy generates its categories.

[TABLE]

• *

category(x, y, contains_x)

Dummy method which needs to be implemented by the inheriting Venn taxonomy.

Computes the category of the example $z:=(\texttt{x},\texttt{y})$.

[TABLE]

• •

vtx.VTXKNearestNeighbors(**sklearn)

Class implementing a Venn taxonomy based on the $k$-nearest neighbors method for classification (see Chapter 5).

The implementation is based on the scikit-learn implementation (see Buitinck et al., 2013).

[TABLE]

Methods

See \hyperref[itm:vtxbase] vtx.base.VTXBase.

CP(A, epsilons, smoothed = False,

mondrian_taxonomy = _not_mcp)

Class implementing conformal prediction (see Chapter 2).

[TABLE]

Methods

• •

train(X, y, override = False)

Wrapper for the train method of the nonconformity measure A (see

\hyperref[itm:ncsbase_train] ncs.base.NCSBase.train).

[TABLE]

• •

predict(X)

[TABLE]

• •

predict_best(X, significance_levels = True)

Method for determining only the best label as prediction.

[TABLE]

• •

score(X, y)

Generates metrics for the predictor.

[TABLE]

• •

score_online(X, y)

Iterates through a set of examples, predicts each example and then trains the predictor with it before predicting the next example. Generates metrics for the predictor.

[TABLE]

• •

p_vals(X)

Computes the p-values for each observation from a bag, paired with each label $y\in\textbf{Y}$.

[TABLE]

ICP(A, epsilons, smoothed = False,

mondrian_taxonomy = _not_mcp)

Class implementing inductive conformal prediction (see Chapter 3).

[TABLE]

Methods

• •

train(X, y, override = False)

See \hyperref[itm:cp_train]CP.train.

• •

calibrate(X, y, override = False)

Method for calibrating the ICP (see Chapter 3).

[TABLE]

• •

predict(X)

See \hyperref[itm:cp_predict] CP.predict.

• •

predict_best(X, p_vals = True)

See \hyperref[itm:cp_predict_best] CP.predict_best.

• •

score(X, y)

See \hyperref[itm:cp_score] CP.score.

• •

p_vals(X)

See \hyperref[itm:cp_p_vals] CP.p_vals.

RRCM(A, epsilons,

convex_hull = True)

Class implementing the ridge regression confidence machine (see Chapter 2.2).

[TABLE]

Methods

• •

train(X, y, override = False)

See \hyperref[itm:cp_train] CP.train.

• •

predict(X)

See \hyperref[itm:cp_predict] CP.predict.

• •

score(X, y)

See \hyperref[itm:cp_score] CP.score.

• •

score_online(X, y)

See \hyperref[itm:cp_score_online] CP.score_online.

Venn(venn_taxonomy)

Class implementing Venn prediction (see Chapter 5).

[TABLE]

Methods

• •

train(X, y, override = False)

See \hyperref[itm:cp_train] CP.train.

• •

predict(X, proba = True)

[TABLE]

• •

score(X, y)

See \hyperref[itm:cp_score] CP.score.

• •

score_online(X, y)

See \hyperref[itm:cp_score_online] CP.score_online.

Meta(M_train, M_predict,

B_train, B_predict, epsilons)

Class implementing meta-conformal prediction (see Chapter 6).

[TABLE]

Methods

• •

train(X, y, k_folds, plot = False)

Generates the internal threshold $T$ and trains $B$ and $M$.

[TABLE]

• •

predict(X)

See \hyperref[itm:cp_predict] CP.predict.

• •

score(X, y)

See \hyperref[itm:cp_score] CP.score.

Appendix B Examples

B.1 CP on the iris data set with

different nonconformity measures

B.1.1 Nonconformity measure based on keras

neural net

B.2 ICP on the iris data set with

different nonconformity measures

B.3 MCP on the iris data set, both

inductive and transductive

B.4 RRCM on the Boston housing data set,

both online and offline

B.5 Venn on the iris data set

B.6 Meta on the iris data set