Functional Correlations in the Pursuit of Performance Assessment of Classifiers

TL;DR

This paper introduces new correlation measures for evaluating classifiers, demonstrating their effectiveness in analyzing confusion matrices and offering novel tools for performance assessment.

Contribution

The paper proposes and explores CO-, ANTI-, and COANTI-correlation coefficients as new measures of association for classifier evaluation.

Findings

New correlation coefficients effectively classify confusion matrices.

The coefficients differ from existing measures in the literature.

Illustrative examples demonstrate their practical utility.

Abstract

In statistical classification and machine learning, as well as in social and other sciences, a number of measures of association have been proposed for assessing and comparing individual classifiers, raters, as well as their groups. In this paper, we introduce, justify, and explore several new measures of association, which we call CO-, ANTI- and COANTI-correlation coefficients, that we demonstrate to be powerful tools for classifying confusion matrices. We illustrate the performance of these new coefficients using a number of examples, from which we also conclude that the coefficients are new objects in the sense that they differ from those already in the literature.

Equations

$$M_{X,Y}=(p_{i,j}{)}_{i,j=1}^d,$$

$$C(\mathbf{f},\mathbf{g})={\sum_{i,j=1}^{d}f_{i}p_{ij}g_{j}-\big{(}\sum_{i=1}^{d}f_{i}p_{i\bullet}\big{)}\big{(}\sum_{j=1}^{d}g_{j}p_{\bullet j}\big{)}\over\sqrt{\sum_{i=1}^{d}f_{i}^{2}p_{i\bullet}-\big{(}\sum_{i=1}^{d}f_{i}p_{i\bullet}\big{)}^{2}}~{}\sqrt{\sum_{j=1}^{d}g_{j}^{2}p_{\bullet j}-\big{(}\sum_{j=1}^{d}g_{j}p_{\bullet j}\big{)}^{2}}}\in[-1,1],$$

$$\varrho (X,Y)=\mathrm{Corr}[X,Y]\in [-1,1]$$

$$\varrho (X,Y\mid \mathcal{A})=\underset{(g,h)\in \mathcal{A}}{\sup }\mathrm{Corr}[g(X),h(Y)]\in [-1,1]$$

$$\mathcal{B}=\big{\{}(g,h)~{}:~{}\mathrm{Var}[g(X)],\mathrm{Var}[h(Y)]\in(0,\infty)\big{\}}.$$

$${\varrho }_{\text{textsc}sup}(X,Y)=\varrho (X,Y\mid \mathcal{B})\in [0,1].$$

$$\mathcal{B}_{1}=\big{\{}(\mathbb{I}_{(-\infty,x]},\mathbb{I}_{(-\infty,y]}\big{)}~{}:~{}x,y\in\mathbb{R},~{}F_{X}(x),F_{Y}(y)\in(0,1)\big{\}}$$

$$\varrho (X,Y\mid \mathcal{A})=0⟺X\perp \text{mspace}-10.0mu\perp Y.$$

$$\mathcal{B}_{0}=\big{\{}(g_{0},h_{0})~{}:~{}\mathrm{Var}[X],\mathrm{Var}[Y]\in(0,\infty)\big{\}}$$

$${\varrho }_{\text{textsc}mon}(X,Y)=\varrho (X,Y\mid {\mathcal{B}}_{\text{textsc}mon})\in [0,1].$$

$$\big{|}\varrho(X,Y)\big{|}\leq\varrho_{\textsc{mon}}(X,Y)\leq\varrho_{\textsc{sup}}(X,Y).$$

$${\varrho }_{\text{textsc}ii}(X,Y)$$

$${\varrho }_{\text{textsc}id}(X,Y)$$

$$\varrho_{\textsc{mon}}(X,Y)=\max\big{\{}\varrho_{\textsc{ii}}(X,Y),\varrho_{\textsc{id}}(X,Y)\big{\}}\in[0,1].$$

$$\big{(}g(x)-g(x^{\prime})\big{)}\big{(}h(x)-h(x^{\prime})\big{)}\geq 0$$

$${\varrho }_{\text{textsc}co}(X,Y)=\varrho (X,Y\mid {\mathcal{B}}_{\text{textsc}co})\in [-1,1]$$

$$\big{(}g(x)-g(x^{\prime})\big{)}\big{(}h(x)-h(x^{\prime})\big{)}\leq 0$$

$${\varrho }_{\text{textsc}anti}(X,Y)=\varrho (X,Y\mid {\mathcal{B}}_{\text{textsc}anti})\in [-1,1].$$

$${\varrho }_{\text{textsc}co}(X,Y)=\underset{h}{\sup }{\varrho }_{\text{textsc}ii}(h(X),h(Y)),$$

$${\varrho }_{\text{textsc}anti}(X,Y)=\underset{h}{\sup }{\varrho }_{\text{textsc}id}(h(X),h(Y)),$$

$${\varrho }_{\text{textsc}coanti}(X,Y)=\varrho (X,Y\mid {\mathcal{B}}_{\text{textsc}coanti})\in [0,1]$$

$$\varrho_{\textsc{coanti}}(X,Y)=\max\big{\{}\varrho_{\textsc{co}}(X,Y),\varrho_{\textsc{anti}}(X,Y)\big{\}},$$

$$\varrho_{\textsc{coanti}}(X,Y)=\max\big{\{}\sup_{h}\varrho_{\textsc{ii}}(h(X),h(Y)),\sup_{h}\varrho_{\textsc{id}}(h(X),h(Y))\big{\}}.$$

$${\varrho }_{\text{textsc}mon}(X,Y)\le {\varrho }_{\text{textsc}coanti}(X,Y)\le {\varrho }_{\text{textsc}sup}(X,Y)$$

$$\big{|}\varrho(X,Y)\big{|}\leq\varrho_{\textsc{mon}}(X,Y)\leq\varrho_{\textsc{coanti}}(X,Y)\leq\varrho_{\textsc{sup}}(X,Y).$$

$${\varrho }_{\text{textsc}coanti}(X,Y)=\underset{h}{\sup }{\varrho }_{\text{textsc}mon}(h(X),h(Y)),$$

$$\sup_{h}\varrho_{\textsc{ii}}(h(X),h(Y))=\sup_{h}\max\big{\{}\varrho_{\textsc{ii}}(h(X),h(Y)),\varrho_{\textsc{id}}(h_{*}(X),h_{*}(Y))\big{\}}.$$

$$\underset{h}{\sup }{\varrho }_{\text{textsc}ii}(h(X),h(Y))$$

$$=\underset{h}{\sup }{\varrho }_{\text{textsc}mon}(h(X),h(Y)),$$

$${\varrho }_{\text{textsc}mon}(X,Y)<{\varrho }_{\text{textsc}coanti}(X,Y)<{\varrho }_{\text{textsc}sup}(X,Y).$$

$$\sum _{i=1}^d{f}_i{p}_{i\bullet }=\sum _{j=1}^d{g}_j{p}_{\bullet j}=0\text{and}\sum _{i=1}^d{f}_i^2{p}_{i\bullet }=\sum _{j=1}^d{g}_j^2{p}_{\bullet j}=1,$$

$$C(\mathbf{f},\mathbf{g})=\sum _{i,j=1}^d{f}_i{p}_{ij}{g}_j.$$

Full text

1 Introduction

Classification has been a massively important and rapidly developing area of research, and for a glimpse of recent developments on the topic, we refer, for example, to selected contributions presented at the two latest scientific meetings of the Classification and Data Analysis Group (CLADAG) of the Italian Statistical Society (e.g., Mola et al., 2018; Greselin et al., 2019, and references therein).

In the present paper, we focus on ordinal classification, which is a fundamental problem in supervised learning. Many classification algorithms, including support vector machines (Vapnik, 2000), random forests (Breiman, 2001), and instance-based methods (e.g., Gweon et al., 2019, and references therein) have been modified to handle problems with ordered classes (Chu and Keerthi, 2007; Hechenbichler and Schliep, 2004; Janitza et al., 2016).

Model assessment is an important task in supervised classification. In binary or multi-class problems, a common approach to model evaluation is to use confusion matrices and calculate accuracy measures such as error rate, precision and recall. Other approaches include graphical assessment methods such as the ROC curve and reliability diagram (Gweon and Yu, 2019). When classification problems contain ordered classes, employing those metrics faces difficulties in evaluating classifiers because they ignore the ordering information in the response variable.

For an effective and consistent evaluation for ordinal classification methods, Cardoso and Pinto da Costa (2007) and Cardoso and Sousa (2011) have employed concordant and discordant vectors. These studies have inspired our current work that explores the potential of using various facets of the notion of comonotonicity (e.g., Wang and Zitikis, 2020, and references therein) for tackling classification problems.

Namely, let $\{1,\dots,d\}$ be the set of ordinal classes to which subjects or objects $\omega\in\Omega$ are classified. For example, we may think of six ($d=6$) education levels: high school, associate’s, bachelor’s, master’s, professional, and doctoral (e.g., Torpey, 2018). Assume that there are two classifiers, $X$ and $Y$, which are mappings from $\Omega$ to $\{1,\dots,d\}$ and can therefore be formally viewed as random variables defined on the sample space $\Omega$. To exemplify, we may think of $X$ and $Y$ as the highest education levels achieved by parents and their offsprings, respectively.

With $p_{i,j}$ denoting the joint probability $\mathbb{P}(X=i,Y=j)$, we obtain the matrix

[TABLE]

which is frequently called the confusion matrix in statistical classification. The marginal distributions of $X$ and $Y$ are given by the probabilities $p_{i\bullet}=\sum_{j=1}^{d}p_{ij}$ and $p_{\bullet j}=\sum_{i=1}^{d}p_{ij}$, respectively. For illustrative examples, we refer to studies of social mobility (e.g., Theil and Stambaugh, 1977; Cox et al., 2009; Beller and Hout, 2006), and problems of psychometry (e.g., Cohen, 1968; Warrens, 2013b). Later in this paper, we shall revisit confusion matrices analyzed by Cardoso and Sousa (2011).

Associated with the classifier $X$, there is a valuation function $f:\{1,\dots,d\}\to\mathbb{R}$. For example, we may think of the parent’s salary $f(X)$ corresponding to the education level $X$. Likewise, associated with the classifier $Y$, there is a (possibly different) valuation function $g:\{1,\dots,d\}\to\mathbb{R}$, and we may think of $g(Y)$ as the offspring’s salary corresponding to the education level $Y$. From these valuations, we obtain the $d$-variate valuation vectors $\mathbf{f}=(f_{i})^{\top}$ and $\mathbf{g}=(g_{i})^{\top}$, where $f_{i}=f(i)$ and $g_{j}=g(j)$. The correlation coefficient between the two valuations with respect to the confusion matrix $M_{X,Y}$ is

[TABLE]

which is well defined whenever $\mathbf{f},\mathbf{g}\in\mathbb{R}^{d}$ are non-degenerate, that is, when the condition $\sum_{i=1}^{d}f_{i}^{2}p_{i\bullet}-(\sum_{i=1}^{d}f_{i}p_{i\bullet})^{2}>0$ holds for $\mathbf{f}$, and an analogous condition holds for $\mathbf{g}$.

The coefficient $C(\mathbf{f},\mathbf{g})$ assesses correlation at the individual level, that is, for each pair $\mathbf{f}$ and $\mathbf{g}$ of valuations, such as, using our earlier illustrative interpretation, the salaries corresponding to the highest education levels $X$ and $Y$ achieved by parents and their offsprings. Note, however, that salaries (usually) vary within certain ranges, depending on the education level. Keeping this in mind, instead of dealing with the correlation $C(\mathbf{f},\mathbf{g})$ for any specific pair of valuations $\mathbf{f}$ and $\mathbf{g}$, we deal with sets of valuations (for example, salary brackets) that satisfy certain order requirements.

For example, assuming that higher levels of education result in higher levels of pay, we naturally deal with “increasing-increasing” (II-, for short) valuations, that is, with those $\mathbf{f}$ and $\mathbf{g}$ for which the inequality constraints $f_{i}\leq f_{j}$ and $g_{i}\leq g_{j}$ hold whenever $1\leq i\leq j\leq d$. The maximal correlation coefficient over this set of valuations $\mathbf{f}$ and $\mathbf{g}$ gives rise to what is known in the literature as the II-correlation coefficient (Kimeldorf and Sampson, 1978; Kimeldorf et al., 1982). Likewise, we have the “increasing-decreasing” (ID-) correlation coefficient. Combining these two coefficients, we obtain the monotone (MON-) correlation coefficient. We refer to Kimeldorf and Sampson (1978), and Kimeldorf et al. (1982) for properties and other details related to these correlation coefficients, but we shall also discuss them later in this paper. Note that when we maximize $C(\mathbf{f},\mathbf{g})$ with respect to all non-degenerate pairs $\mathbf{f}$ and $\mathbf{g}$, we arrive at the supremum (SUP-) correlation coefficient of Gebelein (1941).

Following the terminology of Schmeidler (1986) and Denneberg (1994), we call the vectors $\mathbf{f}$ and $\mathbf{g}$ comonotone when $(f_{i}-f_{j})(g_{i}-g_{j})\geq 0$ for all $1\leq i,j\leq d$, and antimonotone when $(f_{i}-f_{j})(g_{i}-g_{j})\leq 0$ for all $1\leq i,j\leq d$. Maximizing $C(\mathbf{f},\mathbf{g})$ with respect to all non-degenerate comonotone (resp. antimonotone) pairs $(\mathbf{f},\mathbf{g})$ gives rise to what we call comonotone (CO-) and antimonotone (ANTI-) correlation coefficients, respectively. We shall show in the following sections that the latter two coefficients as well as a third one that we call the COANTI-correlation coefficient (to be introduced in Section 3 below) are new notions that, in general, differ from any of the known functional correlation coefficients, such as the aforementioned II-, ID-, and MON-correlation coefficients.

To start appreciating these new coefficients and see their potential in classification, we note that comonotone pairs $(\mathbf{f},\mathbf{g})$ are common and natural valuations in the context of social mobility. For example, referring to the aforementioned example concerning the education levels and salaries, we see from the charts provided by Torpey (2018) that salaries do not always increase when educational levels increase. Specifically, a bar chart reported by Torpey (2018) shows that among the aforementioned six levels of education, the median weekly earnings peek at the penultimate (i.e., professional) degree, thus making the earlier used II-correlation coefficient suboptimal, while the CO-correlation coefficient perfectly suits the purpose.

The rest of the paper is organized as follows. In Section 2, we discuss the classical II-, ID- and MON-correlation coefficients in the case of generic classifiers $X$ and $Y$. In Section 3, we formally define and discuss the already mentioned CO-, ANTI- and COANTI-correlation coefficients, which are new measures of association. In Section 4, we provide a computational method for all (new and old) functional correlation coefficients. In Section 5, we put forward a flowchart for comparing confusion matrices and then use it to revisit matrices of Cardoso and Sousa (2011). We shall see from the numerical results obtained therein that the newly introduced functional correlation coefficients, in conjunction with the classical ones, provide a powerful and unifying approach to classifying confusion matrices. Section 6 concludes the paper with a brief summary of main contributions and some remarks.

2 Classical functional correlation coefficients

The Pearson correlation coefficient

[TABLE]

has played a pivotal role in various research areas. It is not, however, an independence determining coefficient, that is, the equation $\varrho(X,Y)=0$ does not, in general, imply that $X$ and $Y$ are independent, henceforth shorthanded as $X\mathrel{\perp\mspace{-10.0mu}\perp}Y$. To achieve independence determination, the functional correlation coefficient (cf. Sampson, 1992)

[TABLE]

becomes a natural tool, defined for various subsets $\mathcal{A}$ of the set

[TABLE]

Throughout the paper, we deal only with Borel functions, and thus $g$ and $h$ in the definition of $\mathcal{B}$ are tacitly assumed to be such. Note that the set $\mathcal{B}$ depends on the cumulative distribution functions (cdf’s) $F_{X}$ and $F_{Y}$ of $X$ and $Y$, respectively. Obviously, when $\mathcal{A}=\mathcal{B}$, the coefficient $\varrho(X,Y\mid\mathcal{A})$ is the largest and therefore called the supremum (SUP-) correlation coefficient (Gebelein, 1941), defined by the equation

[TABLE]

Note 1

Unless $\mathrm{Var}[X]\in(0,\infty)$ and $\mathrm{Var}[Y]\in(0,\infty)$, the Pearson correlation coefficient $\varrho(X,Y)$ does not exist, but the functional correlation coefficient $\varrho(X,Y\mid\mathcal{B})$ may nevertheless be well defined. For example, when $X=Y\sim\text{Cauchy}(0,1)$, the classical Pearson coefficient $\varrho(X,Y)$ does not exist, but the functional coefficient $\varrho(X,Y\mid\mathcal{B})$ is finite and equal to $1$.

The SUP-correlation coefficient $\varrho_{\textsc{sup}}(X,Y)$ is independence determining because $\mathcal{B}\supseteq\mathcal{B}_{1}$, where

[TABLE]

with $\mathbb{I}_{(-\infty,x]}$ denoting the indicator function of the interval $(-\infty,x]$, that is, $\mathbb{I}_{(-\infty,x]}(z)$ is equal to $1$ when $z\in(-\infty,x]$ and [math] when $z\notin(-\infty,x]$. Indeed, for a set $\mathcal{A}$ to be independence determining, the necessary and sufficient condition is $\mathcal{A}\supseteq\mathcal{B}_{1}$, under which we have the equivalence relationship

[TABLE]

When the set $\mathcal{A}$ is the singleton

[TABLE]

with $g_{0}(x)=x$ and $h_{0}(x)=x$ for all $x\in\mathbb{R}$, coefficient (3) reduces to the Pearson correlation coefficient, that is, $\varrho(X,Y)=\varrho(X,Y\mid\mathcal{B}_{0})$. The singleton $\mathcal{B}_{0}$ is not, however, independence determining, which is of course a well known fact.

When $\mathcal{A}$ is $\mathcal{B}_{\textsc{mon}}$ consisting of all the pairs $(g,h)\in\mathcal{B}$ of monotone functions $g$ and $h$, we have the monotone (MON-) correlation coefficient (Kimeldorf and Sampson, 1978)

[TABLE]

It is an independence determining coefficient because $\mathcal{B}_{\textsc{mon}}\supset\mathcal{B}_{1}$.

Since $\mathcal{B}_{\textsc{mon}}\subset\mathcal{B}$, we have $\varrho_{\textsc{mon}}(X,Y)\leq\varrho_{\textsc{sup}}(X,Y)$. To compare $\varrho_{\textsc{mon}}(X,Y)$ with the Pearson correlation coefficient $\varrho(X,Y)$, we need to assume $\mathrm{Var}[X]\in(0,\infty)$ and $\mathrm{Var}[Y]\in(0,\infty)$. In this case, we have the inclusions $\mathcal{B}_{0}\subset\mathcal{B}_{\textsc{mon}}\subset\mathcal{B}$ and thus the ordering $\varrho(X,Y)\leq\varrho_{\textsc{mon}}(X,Y)\leq\varrho_{\textsc{sup}}(X,Y)$. In fact, the following inequalities hold:

[TABLE]

We can easily find random variables $X$ and $Y$ such that $|\varrho(X,Y)|<\varrho_{\textsc{mon}}(X,Y)$. We also know from Kimeldorf and Sampson (1978) that there are $X$ and $Y$ such that $\varrho_{\textsc{mon}}(X,Y)<\varrho_{\textsc{sup}}(X,Y)$. Hence, in general, the three correlation coefficients in (6) are distinct.

The set $\mathcal{B}_{\textsc{mon}}$ is the union of i) $\mathcal{B}_{\textsc{ii}}$ consisting of the pairs $(g,h)\in\mathcal{B}$ of increasing functions $f$ and $g$, ii) $\mathcal{B}_{\textsc{id}}$ consisting of the pairs $(g,h)\in\mathcal{B}$ of increasing functions $f$ and decreasing functions $g$, and iii) the sets $\mathcal{B}_{\textsc{di}}$ and $\mathcal{B}_{\textsc{dd}}$ defined analogously. Since $\mathrm{Corr}[g(X),h(Y)]$ is equal to $\mathrm{Corr}[-g(X),-h(Y)]$, we have the equations $\varrho(X,Y\mid\mathcal{B}_{\textsc{di}})=\varrho(X,Y\mid\mathcal{B}_{\textsc{id}})$ and $\varrho(X,Y\mid\mathcal{B}_{\textsc{dd}})=\varrho(X,Y\mid\mathcal{B}_{\textsc{ii}})$, which effectively leaves us with only two correlation coefficients (Kimeldorf et al., 1982):

[TABLE]

In the terminology of Kimeldorf et al. (1982), $\varrho_{\textsc{ii}}(X,Y)$ is the concordant monotone correlation coefficient, and $-\varrho_{\textsc{id}}(X,Y)$ is the discordant monotone correlation coefficient. Throughout the paper, we call $\varrho_{\textsc{ii}}(X,Y)$ the II-correlation coefficient and $\varrho_{\textsc{id}}(X,Y)$ the ID-correlation coefficient. Kimeldorf et al. (1982) note the equation

[TABLE]

The three functional correlation coefficients making up equation (7) have played a considerable role in understanding social mobility (e.g., Kimeldorf et al., 1982; Sampson, 1992). While working on the current paper and testing various computational algorithms, we explored a number of social mobility tables, such as those reported by Theil and Stambaugh (1977), Cox et al. (2009), and Beller and Hout (2006). Interestingly, all of those tables have shown that their SUP-correlation coefficients are achieved on increasing pairs $g$ and $h$, thus implying that for those tables, the II-, MON- and SUP-correlation coefficients are the same or, to be precise, coincide at least up to the sixth decimal digit, which is our default computational precision throughout this paper. (Obtaining closed-form formulas for any of the functional correlation coefficients seems to be an impossible task, which is not really needed as far as we can see.) Finally, we note that during our numerical explorations, we found the computational ideas of Kimeldorf et al. (1981) particularly helpful.

3 New comonotonicity-related functional correlation coefficients

As we have noted above, the functional correlation coefficients corresponding to the confusion matrices arising from some intergenerational mobility surveys are maximized on increasing $g$ and $h$. This is not, however, always the case, as we see from the data discussed by Torpey (2018), where comonotone $g$ and $h$ manifest naturally. As a reaction to this fact, we next introduce what we call CO-, ANTI- and COANTI-correlation coefficients, and in Section 5 we shall demonstrate their power when comparing confusion matrices. Since analytical formulas are not available due to the nature of these coefficients, we shall offer two effective computational algorithms in Sections 4 and 5.

The aforementioned three correlation coefficients rely on comonotone functions, which play a prominent role in areas such as quantitative finance, insurance, and economics (e.g., Föllmer and Schied, 2016, and references therein). Namely, functions $g$ and $h$ are said to be comonotone (Schmeidler, 1986; Denneberg, 1994) if

[TABLE]

holds for all $x$ and $x^{\prime}$ in the joint domain of definition of both $g$ and $h$.

To the best of our knowledge, the notion of comonotonicity has not yet been fully utilized in classification problems, with initial impressive hints on the notion’s potential in such problems provided by Cardoso and Pinto da Costa (2007) and Cardoso and Sousa (2011).

Definition 3.1**.**

Let $\mathcal{B}_{\textsc{co}}$ denote the set of all pairs $(g,h)\in\mathcal{B}$ of comonotone functions. We call

[TABLE]

the comonotone (CO-) correlation coefficient.

If non-negativity in condition (8) is replaced by non-positivity, then $g$ and $h$ are said to be antimonotone. That is, $g$ and $h$ are antimonotone functions if

[TABLE]

holds for all $x$ and $x^{\prime}$ in the joint domain of definition of both $g$ and $h$.

Definition 3.2**.**

Let $\mathcal{B}_{\textsc{anti}}$ denote the set of all pairs $(g,h)\in\mathcal{B}$ of antimonotone functions. We call

[TABLE]

the antimonotone (ANTI-) correlation coefficient.

We next relate the CO- and ANTI-correlation coefficients to the classical II- and ID-correlation coefficients.

Proposition 3.3**.**

We have the representations

[TABLE]

where the suprema on the right-hand sides of equations (10) and (11) are taken with respect to all functions $h$ for which the suprema are well defined.

Proof 3.4**.**

Equation (10) follows from the fact that any two functions $f$ and $g$ are comonotone if and only if there is a function $h$ and also two increasing functions $h_{1}$ and $h_{2}$ such that $f(x)=h_{1}(h(x))$ and $g(x)=h_{2}(h(x))$ (Denneberg, 1994, Proposition 4.5(iv), 54–55). Equation (11) follows from equation (10) by observing that $f$ and $g$ are antimonotone if and only if $f$ and $(-g)$ are comonotone.

Definition 3.5**.**

Let $\mathcal{B}_{\textsc{coanti}}$ denote the set of all pairs $(g,h)\in\mathcal{B}$ of comonotone, as well as antimonotone, functions $f$ and $g$, and we thus collectively call them “coanti” functions. In turn, we call

[TABLE]

the COANTI-correlation coefficient.

Analogously to equation (7), we have

[TABLE]

which, in view of representations (10) and (11), gives the equation

[TABLE]

Equation (13) is particularly useful when comparing the COANTI- and MON-correlation coefficients by way of comparing the right-hand sides of equations (13) and (7).

Since $\mathcal{B}_{0}\subset\mathcal{B}_{\textsc{mon}}\subset\mathcal{B}_{\textsc{coanti}}\subset\mathcal{B}$, we have the inequalities

[TABLE]

and also, assuming $\mathbb{E}[X^{2}]\in(0,\infty)$ and $\mathbb{E}[Y^{2}]\in(0,\infty)$,

[TABLE]

Since $\mathcal{B}_{1}\subset\mathcal{B}_{\textsc{mon}}\subset\mathcal{B}_{\textsc{coanti}}\subset\mathcal{B}$, the MON-, COANTI-, and SUP-correlation coefficients are independence determining, that is, equivalence relationship (4) holds with $\mathcal{A}$ replaced by the respective three $\mathcal{B}$’s. The following proposition connects the MON- and COANTI-correlation coefficients.

Proposition 3.6**.**

The COANTI- and MON-correlation coefficients are related via the equation

[TABLE]

where the supremum on the right-hand side of equation (16) is taken with respect to all functions $h$ for which the supremum is well defined.

Proof 3.7**.**

We start with equation (13) and assume for the sake of argument that $\varrho_{\textsc{coanti}}(X,Y)$ is maximized by $\sup_{h}\varrho_{\textsc{ii}}(h(X),h(Y))$. This implies that $\sup_{h}\varrho_{\textsc{ii}}(h(X),h(Y))$ is not smaller than $\varrho_{\textsc{id}}(h_{*}(X),h_{*}(Y))$ for every function $h_{*}$ for which the latter correlation coefficient is well defined. Hence, we have the equation

[TABLE]

Since equation (17) holds for every $h_{*}$, we can choose $h_{*}$ to be $h$. We obtain the equations

[TABLE]

where the right-most equation holds due to equation (7). Hence, $\varrho_{\textsc{coanti}}(X,Y)$ is equal to the right-hand side of equation (7). We also arrive at the same conclusion when $\varrho_{\textsc{coanti}}(X,Y)$ is maximized by $\sup_{h}\varrho_{\textsc{id}}(h(X),h(Y))$, thus finishing the verification of Property 3.6.

It should be noted that we have not yet provided convincing evidence that the three functional correlation coefficients (that is, CO, ANTI, and COANTI) are truly new, that is, that they differ from the classical ones of Section 2. For this reason, in the next section we shall construct a confusion matrix (Example 4.3) that gives rise to the following (strict) inequalities:

[TABLE]

The values of the three correlation coefficients have to be calculated numerically, and we shall do so at the usual for this paper precision of six decimal digits, but we shall see (equation (28) below) differences in their values at the first decimal digit.

We conclude the current section by enhancing our intuitive appreciation of the three functional correlation coefficients in (19). For this, we first recall the six education levels noted earlier: high school, associate’s, bachelor’s, master’s, professional, and doctoral (e.g., Torpey, 2018). Suppose next that we are interested in determining the strongest possible correlation between all the possible salaries of, say, daughters and mothers depending on their education levels.

If we assume that salaries increase when the education level increases, then it is appropriate to use the (classical) coefficient $\varrho_{\textsc{ii}}(X,Y)$ or, more broadly, $\varrho_{\textsc{mon}}(X,Y)$, but as Torpey (2018) illustrates, this assumption of monotonicity is sometimes too optimistic to be realistic.

Allowing for varying salary patterns within the group of daughters, as well as within the group of mothers, gives rise to the need for calculating the above introduced coefficient $\varrho_{\textsc{co}}(X,Y)$ or, more broadly, $\varrho_{\textsc{coanti}}(X,Y)$, which assesses maximal synchronicity (that is, comonotonicity) between the salaries of daughters and mothers, without subjectively imposing the opinion that the salaries are increasing when the education level increases.

Finally, the (classical) correlation coefficient $\varrho_{\textsc{sup}}(X,Y)$ allows not only for various salary patterns within the group of daughters, as well as within the group of mothers, but also various patterns between the two groups. Hence, the coefficient $\varrho_{\textsc{sup}}(X,Y)$ is not designed to assess synchronicity between the two groups – it measures the strongest possible correlation with respect to every possible salary pattern that may arise.

We shall keep these illustrations in mind when assessing and interpreting confusions matrices in the following sections, and in particular when describing the flowchart presented in Section 5.1.

4 Calculating functional correlation coefficients

In this section we work within the framework of Section 1. That is, we let $X$ and $Y$ be two classifiers taking values in the set $\{1,\dots,d\}$. Their joint probabilities $p_{i,j}=\mathbb{P}(X=i,Y=j)$ give rise to confusion matrix (1). The correlation coefficient $C(\mathbf{f},\mathbf{g})$ between the valuation vectors $\mathbf{f}=(f_{i})^{\top}$ and $\mathbf{g}=(g_{i})^{\top}$ is defined by formula (2). Without loss of generality we can, and thus do, impose the constraints

[TABLE]

under which the correlation coefficient simplifies to

[TABLE]

4.1 An excursion into psychometric literature

Equation (21) plays a pivotal role in connecting the current research with a large body of literature dealing with the weighted kappa (Cohen, 1968). Indeed, keeping in mind constraints (20), equation (21) can be rewritten as follows:

[TABLE]

where $w_{ij}=(f_{i}-g_{j})^{2}$. Usually in the literature, the vectors $\mathbf{f}$ and $\mathbf{g}$ are set to $f_{i}=i$ and $g_{j}=j$ for all $1\leq i,j\leq d$. Note that these special choices correspond to the class $\mathcal{B}_{0}$ defined by equation (5). The weights $w_{ij}$ become quadratic $w_{ij}=(i-j)^{2}$ (Fleiss and Cohen, 1973).

The right-hand side of equation (22) defines what is known in the literature as the weighted kappa, with $w_{ij}$’s called “disagreement” weights, which can be any, depending on the problem at hand. For example, Cohen (1960) uses the indicator weights $w_{ij}=\mathbb{I}\{i\neq j\}$. Cicchetti and Allison (1971) suggest to use the “linear” (also known as absolute) weights $w_{ij}=|i-j|$. Analyses and comparisons of linear and quadratic weights have been done by Vanbelle and Albert (2009a), Vanbelle (2016), and Kvålseth (2018), where we also find extensive references to earlier works on the topic.

Weights $w_{ij}$ based on more general valuation vectors $\mathbf{f}$ and $\mathbf{g}$, known as category scores, have also been considered, although quite often by setting $\mathbf{f}=\mathbf{g}$. For example, Von Eye and Mun (2005), Tang et al. (2012), and Kvålseth (2018) use disagreement weights such as $w_{ij}=|f_{i}-f_{j}|$ and $w_{ij}=(f_{i}-f_{j})^{2}$ for monotone $f_{1}\leq\cdots\leq f_{d}$. In this paper, we do not impose requirements such as $\mathbf{f}=\mathbf{g}$, thus allowing for the possibility of having very diverse classification patterns. Yet, as is the case with all indices, by condensing data into one number, the weighted kappa inevitably loses information, irrespective of the weights employed.

Some specific criticism has been directed toward the quadratic weights by Warrens (2013a, b) and Vanbelle (2016), among others. Indeed, there are several reasons for being cautious when using such weights because they, being non-linear, distort distances between categories. Also notably, since the weighted kappa is closely related to the Pearson correlation coefficient, as pointed out by a number of researchers (e.g., Fleiss and Cohen, 1973; Schuster, 2004; Kvålseth, 2018), the kappa is not an independence determining coefficient, and thus gives rise to some difficulties when measuring dependence between classifiers, as we already noted in Section 1. Fortunately, a good way out of the difficulty exists, and this brings us back to the notion of functional correlations.

4.2 Matrix-based counterparts of the classical functional correlations

We start by recalling that $X\sim F_{X}$ and $Y\sim F_{Y}$ are independent if and only if the equation

[TABLE]

holds for all $x,y\in\mathbb{R}$ such that both $F_{X}(x)$ and $F_{Y}(y)$ are in $(0,1)$. Consequently, when assessing association between classifiers, we can use the Pearson correlation coefficient, but we need to calculate it over a sufficiently large set of valuation vectors (category scores) $\mathbf{f}$ and $\mathbf{g}$. This reasoning, though perhaps not always explicitly stated in the literature, has lead researchers (e.g., Schucany and Frawley, 1973; Hollander and Sethuraman, 1978; Kraemer, 1981; Feigin and Alvo, 1986) to the idea of sorting out classification problems with the help of several indices, inncluding the Pearson correlation coefficient, Spearman’s $\rho$, Kendall’s $\tau$, and Spearman’s footrule. A somewhat different, although closer to ours, research path has been taken by Vanbelle and Albert (2009b, c), and we next elaborate on it.

Within the framework of Section 1, all possible paired classifiers, also known as raters, are represented by the pairs $(\mathbf{f},\mathbf{g})$ of $d$-dimensional valuation vectors. Without loss of generality, we assume that constraints (20) are satisfied. We denote the set of all such pairs $(\mathbf{f},\mathbf{g})$ by $\mathcal{S}_{\textsc{sup}}$. The maximal correlation coefficient over the set of all such pairs with respect to the confusion matrix $M_{X,Y}$ defines the SUP-correlation coefficient

[TABLE]

As we have noted earlier, there are good reasons for restricting the pairs $(\mathbf{f},\mathbf{g})\in\mathcal{S}_{\textsc{sup}}$ to only those that satisfy $f_{i}\leq f_{j}$ and $g_{i}\leq g_{j}$ for all $1\leq i\leq j\leq d$. Denote the set of all such pairs by $\mathcal{S}_{\textsc{ii}}$. The corresponding II-correlation coefficient is

[TABLE]

Analogously, the ID-correlation coefficient is given by

[TABLE]

where $\mathcal{S}_{\textsc{id}}$ consists of all the pairs $(\mathbf{f},\mathbf{g})\in\mathcal{S}_{\textsc{sup}}$ such that $f_{i}\leq f_{j}$ and $g_{i}\geq g_{j}$ for all $1\leq i\leq j\leq d$. Below are two notes concerning confusion matrices that maximize the II- and ID-correlation coefficients.

Note 4.1**.**

If the confusion matrix $M_{X,Y}$ is diagonal, that is, $p_{ij}=0$ for all $i\neq j$, then equation (22) implies the formula

[TABLE]

and thus the II-correlation coefficient is equal to 1. This value is achieved on the pair $(\mathbf{f},\mathbf{g})\in\mathcal{S}_{\textsc{ii}}$ with $f_{i}=g_{i}$ for all $1\leq i\leq d$.

Note 4.2**.**

If the confusion matrix $M_{X,Y}$ is anti-diagonal, that is, $p_{ij}=0$ for all $i+j\neq d+1$, then equation (22) implies the formula

[TABLE]

and thus the ID-correlation coefficient is equal to 1. This value is achieved on the pair $(\mathbf{f},\mathbf{g})\in\mathcal{S}_{\textsc{id}}$ with $f_{i}=g_{d-i+1}$ for all $1\leq i\leq d$.

The MON-correlation coefficient is given by the equation

[TABLE]

where $\mathcal{S}_{\textsc{mon}}$ consists of all those pairs $(\mathbf{f},\mathbf{g})\in\mathcal{S}_{\textsc{sup}}$ whose coordinates are either increasing or decreasing, that is, $\mathcal{S}_{\textsc{mon}}$ is the union of the four sets $\mathcal{S}_{\textsc{ii}}$, $\mathcal{S}_{\textsc{id}}$, $\mathcal{S}_{\textsc{di}}$, and $\mathcal{S}_{\textsc{dd}}$. In fact, analogous arguments to those above equation (7) show that we only need to work with the first two subsets, $\mathcal{S}_{\textsc{ii}}$ and $\mathcal{S}_{\textsc{id}}$, thus reducing equation (24) to

[TABLE]

which in turn gives the following equation

[TABLE]

4.3 The new functional correlations adapted to confusion matrices

The three functional correlation coefficients that we introduced above are CO, ANTI, and COANTI. We shall next adapt their definitions to tackle confusion matrices. We begin with the CO-correlation coefficient and have

[TABLE]

where $\mathcal{S}_{\textsc{co}}$ consists of all comonotone pairs $(\mathbf{f},\mathbf{g})\in\mathcal{S}_{\textsc{sup}}$, that is, of those for which the bound $(f_{i}-f_{j})(g_{i}-g_{j})\geq 0$ holds for all $1\leq i,j\leq d$.

The ANTI-correlation coefficient is

[TABLE]

where $\mathcal{S}_{\textsc{anti}}$ consists of all antimonotone pairs $(\mathbf{f},\mathbf{g})\in\mathcal{S}_{\textsc{sup}}$, that is, of those for which the bound $(f_{i}-f_{j})(g_{i}-g_{j})\leq 0$ holds for all $1\leq i,j\leq d$.

Analogously to equation (12), the COANTI-correlation coefficient is the maximum of the CO- and ANTI-correlation coefficients, that is,

[TABLE]

In view of Notes 4.1 and 4.2, if the confusion matrix $M_{X,Y}$ is diagonal, then the CO-correlation coefficient is equal to 1, and if the confusion matrix is anti-diagonal, then the ANTI-correlation coefficient is equal to 1. In both of these cases, as follows from equation (26), the COANTI-correlation coefficient is equal to 1.

To illustrate the functional correlation coefficients, and in particular to show that the new ones are indeed distinct from those already in the literature, we have devised the following example.

Example 4.3**.**

Let a $3\times 3$ confusion matrix be defined as follows:

[TABLE]

With the help of MATLAB’s fmincon function (optimization with constraints of nonlinear multi-argument functions), we obtain the following values:

[TABLE]

In particular, from these results and equations (25) and (26), we have

[TABLE]

Hence, the COANTI-correlation coefficient can indeed differ from the MON- and SUP-correlation coefficients; recall bounds (19) and the discussion around them. As we have already noted above, all our calculations are at the six decimal-digit precision, even though we are reporting only the first four decimal digits. In Figure 1, we have depicted $\mathbf{f}$ and $\mathbf{g}$ corresponding to the ID-, CO-, ANTI- and SUP-correlation coefficients using the straight lines connecting the points $(1,f_{1}),(2,f_{2}),(3,f_{3})$ as well as those connecting the points $(1,g_{1}),(2,g_{2}),(3,g_{3})$.

5 Functional correlation coefficients in action

As noted above, earlier works by statisticians and social scientists produced fairly satisfactory classification results based on the MON-correlation coefficient, but examples such as those by Torpey (2018) initiate a search for more refined tools.

The classical functional correlations (recall Section 2), apparently forgotten for quite some time, were rejuvenated by Cardoso and Pinto da Costa (2007) and Cardoso and Sousa (2011). As natural consequences of these studies, in this paper we have suggested the CO-, ANTI-, and COANTI-correlation coefficients, which alongside the classical coefficients, give rise to an efficient and accurate assessment of confusion matrices. To streamline this task, we next propose a flowchart.

5.1 A flowchart for comparing confusion matrices

We are now in the position to revisit the thirteen confusion matrices of Cardoso and Sousa (2011) by means of the flowchart depicted in Figure 2.

A few remarks concerning the flowchart are in order (recall the notes at the end of Section 3 for enhancing intuition).

First, it is natural to view a classifier superior to, or at least not inferior than, any other if its CO-correlation coefficient is equal to 1, which happens, for example, in the case of the diagonal confusion matrix, as noted below equation (26). Hence, we view a confusion matrix $M$ inferior to another matrix $N$ when $\varrho_{\textsc{co}}(M)<\varrho_{\textsc{co}}(N)$.

Second, it may happen that $\varrho_{\textsc{co}}(M)=\varrho_{\textsc{co}}(N)$. In this case, it is natural to seek help from the ANTI-correlation coefficient. (We should point out that antimonotonicity is not an antonym of comonotonicity: there are vectors which are neither comonotone nor antimonotone.) We view $M$ inferior to $N$ whenever $\varrho_{\textsc{anti}}(M)>\varrho_{\textsc{anti}}(N)$, which is reflected in the second step of the flowchart.

We also encounter confusion matrices for which we have $\varrho_{\textsc{co}}(M)=\varrho_{\textsc{co}}(N)$ and $\varrho_{\textsc{anti}}(M)=\varrho_{\textsc{anti}}(N)$. In this case, we seek help from the classical II- and ID-correlation coefficients, as pointed out in the third and fourth steps of the flowchart, which mimic the first and second steps, respectively.

5.2 The matrices of Cardoso and Sousa (2011) revisited

We start with the matrices $CM_{1}$–$CM_{6}$ (Cardoso and Sousa, 2011, p. 1185). The first pair that we compare are $CM_{1}$ and $CM_{2}$. We have:

[TABLE]

Since $\varrho_{\textsc{co}}(CM_{1})<\varrho_{\textsc{co}}(CM_{2})$, we conclude that $CM_{1}\prec CM_{2}$, which coincides with the conclusion by Cardoso and Sousa (2011).

Following the discussion of Cardoso and Sousa (2011), we next compare $CM_{3}$ and $CM_{4}$, for which we have the following results:

[TABLE]

Since $\varrho_{\textsc{co}}(CM_{3})<\varrho_{\textsc{co}}(CM_{4})$, we have $CM_{3}\prec CM_{4}$, which coincides with the conclusion by Cardoso and Sousa (2011).

We now compare $CM_{5}$ and $CM_{6}$ of Cardoso and Sousa (2011):

[TABLE]

Since $\varrho_{\textsc{co}}(CM_{5})>\varrho_{\textsc{co}}(CM_{6})$, we have $CM_{5}\succ CM_{6}$, which coincides with the conclusion by Cardoso and Sousa (2011).

We next explore $CM_{10}$, $CM_{11}$ and $CM_{12}$ of Cardoso and Sousa (2011, p. 1187):

[TABLE]

According to the flowchart of Figure 2, we first compare the CO-correlation coefficients of the three matrices and conclude that $CM_{10}\succ CM_{11}$, $CM_{11}\prec CM_{12}$, and $CM_{10}\succ CM_{12}$. Hence, $CM_{10}$ exhibits the best result, which coincides with the conclusion of Cardoso and Sousa (2011). Furthermore, $CM_{11}$ exhibits the worst result among the three matrices, which also coincides with the conclusion of Cardoso and Sousa (2011).

Finally, we look at the confusion matrices $CM(A)$–$CM(D)$ of Cardoso and Sousa (2011, p. 1177). The matrices are the most problematic ones for the classifiers of the aforementioned paper, and they are also troublesome for the flowchart of Figure 2. We begin with results:

[TABLE]

For every $M\in\{CM(A),CM(B),CM(C),CM(D)\}$, we have $\varrho_{\textsc{co}}(M)=1$ and thus the first step of the flowchart fails to differentiate between the matrices. The second step singles out the matrix $CM(A)$ as superior because $\varrho_{\textsc{anti}}(CM(A))<\varrho_{\textsc{anti}}(M)$ for all $M\in\{CM(B),CM(C),CM(D)\}$. To compare the performance of the latter three matrices, we first observe that the third step of the flowchart fails to introduce clarity because $\varrho_{\textsc{ii}}(M)=1$ for all the matrices. Hence, we need to move on to the fourth step, which suggests that $CM(D)$ is inferior to both $CM(B)$ and $CM(C)$ because $\varrho_{\textsc{id}}(CM(D))>\varrho_{\textsc{id}}(M)$ for $M\in\{CM(B),CM(C)\}$. The flowchart does not, however, succeed in differentiating between the matrices $CM(B)$ and $CM(C)$ because their respective correlation coefficients coincide. We conclude the discussion concerning the four matrices $CM(A)-CM(D)$ by noting that all the comparisons we have achieved with the help of the functional correlation coefficients and the flowchart of Figure 2 are in line with the findings of Cardoso and Sousa (2011, pp. 1177–1178).

5.3 An alternative computational algorithm

All numerical results of this paper have been double-checked using an alternative numerical procedure, which we describe next. The starting point for the procedure is definition (2) of $C(\mathbf{f},\mathbf{g})$. We simulate two independent $d$-dimensional Gaussian random vectors, both with independent and standard-Gaussian coordinates.

The choice of the $d$-dimensional Gaussian distribution is natural because after the Euclidean normalization of both $\mathbf{f}$ and $\mathbf{g}$, which can always be done without loss of generality in the context of calculating $C(\mathbf{f},\mathbf{g})$, each of the normalized vectors $\mathbf{f}/\|\mathbf{f}\|$ and $\mathbf{g}/\|\mathbf{g}\|$ uniformly fill in the unit sphere $S^{d-1}$ (e.g., Marsaglia, 1972).

These Gaussian vectors play the roles of $\mathbf{f}$ and $\mathbf{g}$: if they belong to an appropriate set $\mathcal{S}$ (for example, $\mathcal{S}_{\textsc{ii}}$ when calculating $\varrho_{\textsc{ii}}(M_{X,Y})$), then we calculate $C(\mathbf{f},\mathbf{g})$ and record its value. We repeat the procedure as many times as needed to get $10^{6}$ values of $C(\mathbf{f},\mathbf{g})$. Finally, we find the maximal value among the obtained $10^{6}$ values and declare it an approximate value of the functional correlation coefficient under consideration (for example, $\varrho_{\textsc{ii}}(M_{X,Y})$).

6 Concluding notes

In this paper we have recalled several classical (e.g., Sampson, 1992, and references therein) and also introduced and justified new functional correlation coefficients (CO, ANTI, and COANTI) to aid the assessment of confusion matrices. We have explored properties of these coefficients and illustrated their performance using a number of confusion matrices (Cardoso and Sousa, 2011). Our suggested classification algorithm, in the form of a flowchart, is based entirely on these functional correlation coefficients, and it has reached the same conclusions as those by Cardoso and Sousa (2011), who have used a number of diverse (dis)similarity indices.

Finally, we note that the proposed flowchart in Figure 2, with its specific choices and arrangement of functional correlation coefficients, should not be viewed as definitive. Indeed, even though we have used, for example, the CO-correlation coefficient at the very top of the flowchart, due to our subject-matter knowledge gained from the research by, among others, Cardoso and Pinto da Costa (2007), Cardoso and Sousa (2011), and Torpey (2018), there might be contexts, such as classification problems related to social mobility (e.g., Beller and Hout, 2006; Cox et al., 2009), where placing the II- or some other functional correlation coefficient at the very top of the flowchart would be more appropriate.

Acknowledgements

We are grateful to Managing Editor Gernot A. Fink, an Associate Editor, and all the reviewers for the many comments, suggestions, and constructive criticism that guided our work on the revision, which has been substantial. Francesca Greselin’s interest in, and enthusiasm about, classification and related problems have been contagious. Conversations with Ruodu Wang on various facets and applications of comonotonicity have been enlightening. We are indebted to Hyukjun (Jay) Gweon for the many fruitful discussions that we have had on the topic of this paper, as well as for his insights and suggestions related to our results.

Research of the second author has been supported by the Natural Sciences and Engineering Research Council of Canada under the title “From Data to Integrated Risk Management and Smart Living: Mathematical Modelling, Statistical Inference, and Decision Making,” as well as by a Mitacs Accelerate Award from the national research organization Mathematics of Information Technology and Complex Systems, Canada, in partnership with Unilever Canada Inc., under the title “Price Elasticities and Promotion Cannibalization Effect on Promotion Activities.”