Probabilistic Class-Specific Discriminant Analysis

TL;DR

This paper introduces a probabilistic model for class-specific discriminant analysis that captures multi-modal negative class structures and extends existing methods into probabilistic classifiers, improving verification and classification tasks.

Contribution

It formulates a novel probabilistic framework for class-specific discriminant analysis, accommodating multi-modal negative classes and enabling probabilistic classification rules.

Findings

Model captures multi-modal negative class structure.

Existing methods are special cases of the proposed model.

Improved performance in verification and classification tasks.

Abstract

In this paper we formulate a probabilistic model for class-specific discriminant subspace learning. The proposed model can naturally incorporate the multi-modal structure of the negative class, which is neglected by existing class-specific methods. Moreover, it can be directly used to define a class-specific probabilistic classification rule in the discriminant subspace. We show that existing class-specific discriminant analysis methods are special cases of the proposed probabilistic model and, by casting them as probabilistic models, they can be extended to class-specific classifiers. We illustrate the performance of the proposed model in both verification and classification problems.

Figures

Tables

Table 1. TABLE I: Datasets information.

Dataset	$D$	$\#$Samples	$\#$Classes
Jaffe [31]	1200	210	7
Kanade [32]	1200	245	7
BU [33]	1200	700	7
YALE [34]	1200	2432	38
AR [35]	1200	2600	100
15 scenes [36]	512	4485	15
OptDigits [37]	54	5620	10
Caltech101 [38]	512	9145	102

Table 2. TABLE II: Performance on retrieval problems.

	RR	SVR	PCA	LDA	PCSDA-1	PCSDA-K
Jaffe	0.9024 (0.0935)	0.8954 (0.1179)	0.3010 (0.1383)	0.9052 (0.0990)	0.9099 (0.0953)	0.7591 (0.1728)
Kanade	0.6523 (0.2635)	0.6525 (0.2625)	0.2930 (0.1139)	0.6336 (0.2476)	0.6621 (0.2573)	0.4196 (0.2629)
BU	0.6848 (0.2279)	0.6976 (0.2346)	0.2903 (0.1455)	0.6491 (0.2099)	0.6840 (0.1940)	0.6520 (0.1902)
YALE	0.9874 (0.0250)	0.9883 (0.0231)	0.5580 (0.0410)	0.9864 (0.0251)	0.9864 (0.0251)	0.9911 (0.0189)
AR	0.9990 (0.0057)	0.9976 (0.0109)	0.8642 (0.0113)	0.9988 (0.0061)	0.9986 (0.0071)	0.9959 (0.0194)
15 scenes	0.9340 (0.0409)	0.9403 (0.0387)	0.6161 (0.1805)	0.9160 (0.0587)	0.9162 (0.0602)	0.9127 (0.0549)
OptDigits	0.9968 (0.0036)	0.9969 (0.0044)	0.8337 (0.1352)	0.9974 (0.0033)	0.9970 (0.0034)	0.9945 (0.0064)
Caltech101	0.8708 (0.1383)	0.8792 (0.1301)	0.2239 (0.2000)	0.8485 (0.1513)	0.8467 (0.1514)	0.8523 (0.1486)

Table 3. TABLE III: Performance on classification problems.

	RR	SVM	PCA+SVM	LDA+NCC	PCSDA-1	PCSDA-K
Jaffe	0.6750 (0.1032)	0.6714 (0.1225)	0.2809 (0.0542)	0.7049 (0.1299)	0.6261 (0.1262)	0.5242 (0.1249)
Kanade	0.4001 (0.1520)	0.4122 (0.1271)	0.2597 (0.0464)	0.5194 (0.2677)	0.4546 (0.2427)	0.3861 (0.1894)
BU	0.4463 (0.0819)	0.4562 (0.1391)	0.2592 (0.0180)	0.5632 (0.1706)	0.5731 (0.1288)	0.5431 (0.1793)
YALE	0.9369 (0.0090)	0.9396 (0.0851)	0.2715 (0.0851)	0.9517 (0.0208)	0.9409 (0.0214)	0.9611 (0.0222)
AR	0.9664 (0.0352)	0.9683 (0.0343)	0.7346 (0.0394)	0.9760 (0.0452)	0.9674 (0.0481)	0.9516 (0.0629)
15 scenes	0.8997 (0.0854)	0.9005 (0.0517)	0.5365 (0.1507)	0.8993 (0.0539)	0.8573 (0.0654)	0.8130 (0.1275)
OptDigits	0.9875 (0.0900)	0.9881 (0.0082)	0.8985 (0.0706)	0.9899 (0.0077)	0.9569 (0.0141)	0.9826 (0.0074)
Caltech101	0.8130 (0.0971)	0.8140 (0.0807)	0.6042 (0.0394)	0.8075 (0.1448)	0.7605 (0.1829)	0.7663 (0.1955)

Equations

$$\mathcal{J}(\mathbf{W})=\frac{Tr\left({\mathbf{W}}^T{\mathbf{S}}_n\mathbf{W}\right)}{Tr\left({\mathbf{W}}^T{\mathbf{S}}_p\mathbf{W}\right)},$$

$${\mathbf{S}}_n=\sum _{{\mathbf{x}}_i\in {\mathcal{S}}_n}\left({\mathbf{x}}_i-\mathbf{m}\right){\left({\mathbf{x}}_i-\mathbf{m}\right)}^T$$

$${\mathbf{S}}_p=\sum _{{\mathbf{x}}_i\in {\mathcal{S}}_p}\left({\mathbf{x}}_i-\mathbf{m}\right){\left({\mathbf{x}}_i-\mathbf{m}\right)}^T$$

$$P(\mathbf{x}|c_p)\sim N(\mathbf{x}|\mathbf{m},{\mathbf{\Phi }}_p).$$

$$P(\mathbf{y})\sim N(\mathbf{y}|\mathbf{m},{\mathbf{\Phi }}_{\mathbf{n}}),$$

$$P(\mathbf{x}|c_n)=\int N(\mathbf{y}|\mathbf{m},{\mathbf{\Phi }}_n)N(\mathbf{x}|\mathbf{y},{\mathbf{\Phi }}_w)d\mathbf{y},$$

$$P({\mathcal{S}}_p|c_p)=\prod _{i=1}^{N_p}P({\mathbf{x}}_{\mathbf{i}}|c_p).$$

$$P({\mathcal{S}}_k|c_n)=\int N(\mathbf{y}|\mathbf{m},{\mathbf{\Phi }}_n)\prod _{{\mathbf{x}}_i\in {\mathcal{S}}_k}P({\mathbf{x}}_i|\mathbf{y},{\mathbf{\Phi }}_w)d\mathbf{y}.$$

$$P({\mathcal{S}}_p|c_p)=\frac{1}{(2\pi {)}^{\frac{N_{p}D}{2}}|{\mathbf{\Phi }}_p{|}^{\frac{N_p}{2}}}exp\left(-\frac{1}{2}Tr({\mathbf{\Phi }}_p^{-1}{\mathbf{S}}_p)\right)$$

$$P({\mathcal{S}}_k|c_n)$$

$$\mathcal{L}=\ln P({\mathcal{S}}_p|c_p)+\ln P({\mathcal{S}}_n|c_n),$$

$$\ln P({\mathcal{S}}_p|c_p)=C_1-\frac{N_p}{2}\ln |{\mathbf{\Phi }}_p|-\frac{1}{2}Tr({\mathbf{\Phi }}_p^{-1}{\mathbf{S}}_p)$$

$$\ln P({\mathcal{S}}_n|c_n)$$

$$\mathcal{L}$$

$$\frac{\vartheta \mathcal{L}}{\vartheta {\mathbf{\Phi }}_p}=0$$

$$\frac{\vartheta \mathcal{L}}{\vartheta {\mathbf{\Phi }}_n}=0$$

$$\frac{\vartheta \mathcal{L}}{\vartheta {\mathbf{\Phi }}_w}=0$$

$${\mathbf{\Phi }}_n=\frac{1}{K}{\mathbf{S}}_n-\frac{1}{M(N_n-K)}{\mathbf{S}}_w.$$

$${\mathbf{\Phi }}_O={\mathbf{\Phi }}_n+{\mathbf{\Phi }}_w=\frac{1}{K}{\mathbf{S}}_n+\frac{1}{N_n}{\mathbf{S}}_w,$$

$${\mathbf{S}}_n\mathbf{w}=\lambda \left({\mathbf{S}}_p+{\mathbf{S}}_w\right)\mathbf{w}.$$

$${\tilde{\mathbf{\Phi }}}_p={\mathbf{W}}^T{\mathbf{\Phi }}_p\mathbf{W}\text{and}{\tilde{\mathbf{\Phi }}}_O={\mathbf{W}}^T{\mathbf{\Phi }}_O\mathbf{W}.$$

$$P(c_p|{\mathbf{x}}^{\ast })=\frac{p({\mathbf{x}}^{\ast }|c_p)P(c_p)}{p({\mathbf{x}}^{\ast })}$$

$$P(c_n|{\mathbf{x}}^{\ast })=\frac{p({\mathbf{x}}^{\ast }|c_n)P(c_n)}{p({\mathbf{x}}^{\ast })}.$$

$$p({\mathbf{x}}^{\ast }|c_p)$$

$$p({\mathbf{x}}^{\ast }|c_n)$$

$$\lambda ({\mathbf{z}}^{\ast })=\frac{P(c_p)|{\tilde{\mathbf{\Phi }}}_O{|}^{\frac{1}{2}}}{P(c_n)|{\tilde{\mathbf{\Phi }}}_p{|}^{\frac{1}{2}}}\frac{exp\left(-\frac{1}{2}{\mathbf{z}}^{\ast T}{\tilde{\mathbf{\Phi }}}_p^{-1}{\mathbf{z}}^{\ast }\right)}{exp\left(-\frac{1}{2}{\mathbf{z}}^{\ast T}{\tilde{\mathbf{\Phi }}}_O^{-1}{\mathbf{z}}^{\ast }\right)}.$$

$$g({\mathbf{z}}^{\ast })$$

$${\mathbf{S}}_n=\sum _{k=1}^K({\bar{\mathbf{x}}}_k-\mathbf{m})({\bar{\mathbf{x}}}_k-\mathbf{m}{)}^T=\mathbf{X}{\mathbf{L}}_n{\mathbf{X}}^T,$$

$${\mathbf{S}}_I={\mathbf{S}}_p+{\mathbf{S}}_w=\mathbf{X}{\mathbf{L}}_I{\mathbf{X}}^T$$

$${\mathbf{L}}_n=\sum _{k=1}^K\left(\frac{1}{N_k^2}{1}_k{1}_k^T-\frac{1}{N_k{N}_p}(1_k{1}_p^T+1_p{1}_k^T)+\frac{1}{N_p^2}{1}_p{1}_p^T\right)$$

$${\mathbf{L}}_I=\left({\mathbf{J}}_p-\frac{1}{N_p}{1}_p{1}_p^T\right)+\sum _{k=1}^K\left({\mathbf{J}}_k-\frac{1}{N_k}{1}_k{1}_k^T\right).$$

Full text

I Introduction

Class-Specific Discriminant Analysis (CSDA) [1, 2, 3, 4], determines an optimal subspace suitable for verification problems, where the objective is the discrimination of the class of interest from the rest of the world. As an example, let us consider the person identification problem, either through face verification [5], or through exploiting appearance or movement information [6, 7]. Different from person recognition, which is a multi-class classification problem assigning a sample (facial image or movement sequence) to a class in a pre-defined set of classes (person IDs in this case), person identification discriminates the person of interest from all rest people.

While multi-class discriminant analysis models, like Linear Discriminant Analysis (LDA) and its variants [8, 9, 10, 11, 12, 13] can be applied in such problems, they are inherently limited by the adopted class discrimination definition. That is, the maximal dimensionality of the resulting subspace is restricted by the number of classes, due to the rank of the between-class scatter matrix. In verification problems involving two classes LDA and its variants lead to one-dimensional subspaces. On the other hand, CSDA by expressing class discrimination using the out-of-class and intra-class scatter matrices is able to define subspaces the dimensionality of which is restricted by the the number of samples forming the smallest class (which is usually the class of interest) or the number of original space dimensions. By defining multiple discriminant dimensions, CSDA has been shown to achieve better class discrimination and better performance in verification problems compared to LDA [1, 2].

While the definition of class discrimination in CSDA and its variants based on the intra-class and out-of-class scatter matrices overcomes the limitations of LDA related to the dimensionality of the discriminant subspace, it overlooks the structure of the negative class. Since in practice samples forming the negative class belong to many classes, different from the positive one, it is expected that they will form subclasses, as illustrated in Figure 1. Class discrimination as defined by CSDA and its variants disregards this structure. Related research in multi-class discriminant analysis indicates that exploitation of subclass information can enhance discrimination power [14, 15, 16, 17].

In this paper, we formulate the class-specific discriminant analysis optimization problem based on a probabilistic model that incorporates the above-described structure of the negative class. We show that the optimization criterion used by standard CSDA and its variants corresponds to a special case of the proposed probabilistic model, while new discriminant subspaces can be obtained by allowing samples of the negative class to form subclasses automatically determined by applying (unsupervised) clustering techniques on the negative class data. Moreover, the use of the proposed probabilistic model for class-specific discriminant learning naturally leads to a classification rule in the discriminant subspace, something that is not possible when the standard CSDA criterion is considered.

II Related Work

Let us denote by $\mathcal{S}_{p}=\{\mathbf{x}_{1},\dots,\mathbf{x}_{N_{p}}\}$ a set of $N_{p}$ $D$-dimensional vectors representing samples of the positive class and by $\mathcal{S}_{n}=\{\mathbf{x}_{N_{p}+1},\dots,\mathbf{x}_{N}\}$, where $N=N_{p}+N_{n}$, a set of $N_{n}$ vectors representing samples of the negative class. In the following we consider the linear class-specific subspace learning case and we will describe how to perform nonlinear (kernel-based) class-specific subspace learning following the same processing steps in Section III-D. We would like determine a linear projection $\mathbf{W}\in\mathbb{R}^{D\times d}$, mapping $\mathbf{x}_{i}$ to a $d$-dimensional subspace, i.e. $\mathbf{z}_{i}=\mathbf{W}^{T}\mathbf{x}_{i}$ that enhances discrimination of the two classes.

Class-specific Discriminant Analysis defines the projection matrix $\mathbf{W}$ as the one maximizing the following criterion:

[TABLE]

where $Tr(\cdot)$ is the trace operator. $\mathbf{S}_{n}\in\mathbb{R}^{D\times D}$ and $\mathbf{S}_{p}\in\mathbb{R}^{D\times D}$ are the out-of-class and intra-class scatter matrices:

[TABLE]

[TABLE]

where $\mathbf{m}$ is the mean vector of the positive class, i.e. $\mathbf{m}=\frac{1}{N_{p}}\sum_{\mathbf{x}_{i}\in\mathcal{S}_{p}}\mathbf{x}_{i}$. $\mathbf{W}$ is obtained by solving the generalized eigen-analysis problem of $\mathbf{S}_{n}\mathbf{w}=\lambda\mathbf{S}_{p}\mathbf{w}$ and keeping the eigenvectors corresponding to the $d$ largest eigenvalues [18]. In the case where $\mathbf{S}_{n}$ is singular, a regularized version of the above problem is solved.

A Spectral Regression [19] based solution of (1) has been proposed in [1, 2]. Let us denote by $\mathbf{w}$ an eigenvector of the generalized eigen-analysis problem $\mathbf{S}_{n}\mathbf{w}=\lambda\mathbf{S}_{p}\mathbf{w}$ with eigenvalue $\lambda$. By setting $\mathbf{X}^{T}\mathbf{w}=\mathbf{v}$ ($\mathbf{X}$ being the data matrix), the original eigen-analysis problem can be transformed to the following eigen-analysis problem $\mathbf{P}_{n}\mathbf{v}=\lambda\mathbf{P}_{p}\mathbf{v}$, where $\mathbf{P}_{n}=\mathbf{e}_{n}\mathbf{e}_{n}^{T}-\frac{1}{N_{p}}\mathbf{e}_{n}\mathbf{e}_{p}^{T}-\frac{1}{N_{p}}\mathbf{e}_{p}\mathbf{e}_{n}^{T}+\frac{1}{N_{p}^{2}}\mathbf{e}_{p}\mathbf{e}_{p}^{T}$ and $\mathbf{P}_{p}=(1-\frac{2}{N_{p}}+\frac{1}{N_{p}^{2}})\mathbf{e}_{p}\mathbf{e}_{p}^{T}$. Here $\mathbf{e}_{p}\in\mathbb{R}^{N}$ and $\mathbf{e}_{n}\in\mathbb{R}^{N}$ are the positive and negative class indicator vectors. $\mathbf{W}$ is then obtained by applying a two-step process:

• •

Solution of the eigen-analysis problem $\mathbf{P}_{n}\mathbf{v}=\lambda\mathbf{P}_{p}\mathbf{v}$ to determine the matrix $\mathbf{V}=[\mathbf{v}_{1},\dots,\mathbf{v}_{d}]$, where $\mathbf{v}_{i}$ is the eigenvector corresponding to the $i$-th largest eigenvalue.

• •

Calculation of $\mathbf{w}_{i},\>i=1,\dots,d$ such that $\mathbf{X}^{T}\mathbf{w}_{i}=\mathbf{v}_{i}$.

It has been shown in [20] that the generalized eigenvectors $\mathbf{v}$ of $\mathbf{P}_{p}$ and $\mathbf{P}_{n}$ can be directly obtained using the labeling information of the training vectors. However, in that case the order of the eigenvectors is not related to their discrimination ability. It has been also shown in [21] that the class-specific discriminant analysis problem in (1) is equivalent to a low-rank regression problem in which the target vectors are the same as those defined in [20], providing a new proof of the equivalence between class-specific discriminant analysis and class-specific spectral regression.

After determining the data projection matrix $\mathbf{W}$, the training vectors $\mathbf{x}_{i},\>i=1,\dots,N$ are mapped to the discriminant subspace $\mathbf{z}_{i}=\mathbf{W}^{T}\mathbf{x}_{i}$. When a classification problem is considered, a classifier is trained using $\mathbf{z}_{i}$. For example, [2] trains a linear SVM on vectors \mathbf{d}_{i}=|\mathbf{z}_{i}-\mbox{\boldmath\mu}|, where \mbox{\boldmath\mu}=\mathbf{W}^{T}\mathbf{m} and the absolute value operator is applied element-wise on the resulting vector. Due to the need of training an additional classifier, class-specific discriminant analysis models are usually employed in class-specific ranking settings, where test vectors $\mathbf{x}^{*}_{j}$ are mapped to the discriminant subspace $\mathbf{z}^{*}_{j}=\mathbf{W}^{T}\mathbf{x}^{*}_{j}$ and are subsequently ordered based on their distance w.r.t. the positive class mean d_{j}=\|\mathbf{z}^{*}_{j}-\mbox{\boldmath\mu}\|_{2}.

III Probabilistic Class-Specific Learning

In this section, we follow a probabilistic approach for defining a class-specific discrimination criterion that is able to encode subclass information of the negative class. We call the proposed method Probabilistic Class-Specific Discriminant Analysis (PCSDA). PCSDA defines a subspace $\mathbb{R}^{d},\>d\leq D$ of a feature space $\mathbb{R}^{D}$ such that the positive class is optimally discriminated from the negative class, based on the assumption that the negative class is formed by multiple subclasses having the same cardinality. We show how to relax this assumptions in the next subsection.

Let us denote by $\mathbf{x}\in\mathbb{R}^{D}$ a random variable, realizations of which correspond to samples of the positive and negative classes in our problem. PCSDA assumes there exists a positive class $c_{p}$ following a multivariate Gaussian distribution defined by the mean vector $\mathbf{m}\in\mathbb{R}^{D}$ and the covariance matrix $\mathbf{\Phi}_{p}\in\mathbb{R}^{D\times D}$, i.e.:

[TABLE]

Since the negative class $c_{n}$ is formed by samples belonging to multiple classes (which are not distinguished by the labeling information available during training), PCSDA assumes that the negative class forms subclasses, each of which follows a multivariate Gaussian distribution in $\mathbb{R}^{D}$. Let $\mathbf{y}\in\mathbb{R}^{D}$ be a random variable a realization of which corresponds to the mean vector of a negative subclass. Since in class-specific learning we are interested in maximizing the scatter of the negative data from the positive class (represented by $\mathbf{m}$), we model the distribution of $\mathbf{y}$ with respect to $\mathbf{m}$ as a multivariate Gaussian distribution:

[TABLE]

where $\mathbf{\Phi_{n}}$ is the corresponding covariance matrix expressing the scatter of the negative subclasses with respect to the positive class mean $\mathbf{m}$. Then, the distribution of the samples from the negative subclasses with respect to the positive class is expressed by the following multi-modal distribution [22]:

[TABLE]

where $\mathbf{\Phi}_{w}$ is the (common) covariance matrix of the negative subclasses.

III-A Training phase

Given a set of positive i.i.d. samples $\mathcal{S}_{p}=\{\mathbf{x}_{i},\dots,\mathbf{x}_{N_{p}}\}$ the probability of correct assignment under our model is:

[TABLE]

Let us assume that negative class is formed by samples $\mathbf{x}_{i},\>i=N_{p}+1,\dots,N$ belonging to $K$ subclasses, i.e. $\mathcal{S}_{n}=\{\mathcal{S}_{1},\dots,\mathcal{S}_{K}\}$, each having a cardinality of $M=N_{n}/K$. The probability of assigning each of the negative samples to the corresponding negative subclass is given by:

[TABLE]

Without loss of generality, we assume that $\mathbf{m}=\mathbf{0}$ (this can always be done by setting $\mathbf{x}_{i}-\mathbf{m}\rightarrow\mathbf{x}_{i}$). Then:

[TABLE]

[TABLE]

In the above, $\mathbf{S}_{p}$ is the scatter matrix of the positive class, i.e. $\mathbf{S}_{p}=\sum_{\mathbf{x}_{i}\in\mathcal{S}_{p}}\mathbf{x}_{i}\mathbf{x}_{i}^{T}$, $\mathbf{S}_{w}^{(k)}$ is the within-subclass scatter matrix of the $k$-th negative subclass, i.e. $\mathbf{S}_{w}^{(k)}=\sum_{\mathbf{x}_{i}\in\mathcal{S}_{k}}(\mathbf{x}_{i}-\mathbf{\bar{x}}_{k})(\mathbf{x}_{i}-\mathbf{\bar{x}}_{k})^{T}$ and $\mathbf{S}_{n}^{(k)}$ is the scatter matrix of the $k$-th subclass w.r.t. the mean vector of the positive class $\mathbf{m}=\mathbf{0}$, i.e. $\mathbf{S}_{n}^{(k)}=\mathbf{\bar{x}}_{k}\mathbf{\bar{x}}_{k}^{T}$, where $\mathbf{\bar{x}}_{k}$ is the mean vector of the $k$-th negative subclass, i.e. $\mathbf{\bar{x}}_{k}=\frac{1}{M}\sum_{\mathbf{x}_{i}\in\mathcal{S}_{k}}\mathbf{x}_{i}$. Derivations leading to the above results can be found in the supplementary document. Since the assignment of the negative samples to subclasses is not provided by the labels used during training, we define them by applying a clustering method (e.g. $K$-Means) on the negative class vectors.

The parameters of the proposed PCSDA are the covariance matrices $\mathbf{\Phi}_{p}$, $\mathbf{\Phi}_{n}$ and $\mathbf{\Phi}_{w}$ defining the data generation processes for the positive and negative classes. These parameters are estimated by maximizing the (log) probability of correct assignment of the training samples $\mathbf{x}_{i},\>i=1,\dots,N$:

[TABLE]

where

[TABLE]

and

[TABLE]

where $\mathbf{S}_{n}$ and $\mathbf{S}_{w}$ are the total scatter matrices of the negative subclasses, i.e., $\mathbf{S}_{w}=\sum_{k=1}^{K}\mathbf{S}_{w}^{(k)}$ and $\mathbf{S}_{n}=\sum_{k=1}^{K}\mathbf{S}_{n}^{(k)}$.

By substituting (12) and (13) in (11) the optimization problem takes the form:

[TABLE]

The saddle points of $\mathcal{L}$ with respect to $\mathbf{\Phi}_{p}$, $\mathbf{\Phi}_{n}$, $\mathbf{\Phi}_{w}$ lead to:

[TABLE]

Note that when $N_{n}=K$ (17) takes an indeterminant form (i.e. $0/0$). However, in this case each negative subclass is formed by one sample and by definition $\mathbf{\Phi}_{w}=\mathbf{0}$. This is discussed more in subsection III-D. Combining (16) and (17) we get:

[TABLE]

By combining (17) and (18) we can define the following matrix:

[TABLE]

Thus, as can be seen from (15), (17), (18) and (19), the parameters of PCSDA can be calculated using $\mathbf{S}_{p}$, $\mathbf{S}_{n}$ and $\mathbf{S}_{w}$ defined on the training vectors $\mathbf{x}_{i},\>i=1,\dots,N$.

Using $\mathbf{S}_{p}$, $\mathbf{S}_{n}$ and $\mathbf{S}_{w}$ a data transformation $\mathbf{W}\in\mathbb{R}^{D\times D}$ can be obtained by optimizing for:

[TABLE]

Problem (20) corresponds to the generalized eigen-analysis problem of the matrices $\mathbf{S}_{n}$ and $\mathbf{S}_{I}=\mathbf{S}_{p}+\mathbf{S}_{w}$. Since the rank of $\mathbf{S}_{n}$ is $K$, the dimensionality of the obtained subspace is restricted to $d\leq\textrm{min}(D,K)$. Eigenvectors in PCSDA define a data transformation that minimizes the intra-class variance of the positive class and the intra-cluster variance of the negative subclasses, while at the same time maps the means of the negative subclasses as far as possible from the positive class mean. Since the above-described property is optimized by treating (20) as maximization problems, the eigenvectors forming $\mathbf{W}$ are sorted in a decreasing order of the corresponding eigenvalues.

After obtaining $\mathbf{W}$, the intra-class and out-of-class covariance matrices of the transformed data $\mathbf{z}_{i}=\mathbf{W}^{T}\mathbf{x}_{i}$ are given by:

[TABLE]

The process followed in the training phase of PCSDA is illustrated in Algorithm 1.

In the above we set the assumption that the number of samples forming the negative subclasses is equal. This assumption can be relaxed following one of the following approaches. After assigning all negative samples to subclasses and calculating the negative subclass distributions, one can sample $M$ vectors from each distribution. Alternatively, one can calculate the total within-subclass matrix of the negative class by $\mathbf{S}_{w}=\sum_{k=1}^{K}\mathbf{S}_{w}^{(k)}$. The latter approach is commonly used in multi-class discriminant analysis variants [9]. Note that for the model’s parameters calculation, only the overall scatter matrices $\mathbf{S}_{p}$, $\mathbf{S}_{w}$ and $\mathbf{S}_{n}$ are used.

III-B Test phase

After the estimation of the model’s parameters, a new sample $\mathbf{x}^{*}$ can be evaluated. The posterior probabilities of the positive and negative classes are given by:

[TABLE]

[TABLE]

The a priori class probabilities $P(c_{p})$ and $P(c_{n})$ can be calculated by the proportion of the positive and negative samples in the training phase, i.e. $P(c_{p})=N_{p}/N$ and $P(c_{n})=N_{n}/N$. Depending on the problem at hand, it may be sometimes preferable to consider equiprobable classes, i.e. $P(c_{p})=P(c_{n})=1/2$, leading to the maximum likelihood classification case. The class-conditional probabilities of the transformed sample $\mathbf{z}^{*}=\mathbf{W}^{T}\mathbf{x}^{*}$ are given by:

[TABLE]

and

[TABLE]

In the above, we used the orthogonal property of the matrix $\mathbf{W}$. In the case where we want $\mathbf{z}^{*}\in\mathbb{R}^{d},\>d<D$, we keep the dimensions of $\mathbf{z}^{*}$ corresponding to the first $d$ columns of $\mathbf{W}$.

Combining (22)-(25) the ratio of class posterior probabilities is equal to:

[TABLE]

$\lambda(\mathbf{z}^{*})$ can be used to classify $\mathbf{z}^{*}$ to the positive class when $\lambda(\mathbf{z}^{*})>1$ and to the negative class otherwise. Alternatively, (26) can be also used to define the classification rule:

[TABLE]

classifying $\mathbf{z}^{*}$ to the positive class if $g(\mathbf{z}^{*})>0$ and to the negative class otherwise.

In class-specific ranking settings one can follow the process applied when using the standard CSDA approach. First, test vectors $\mathbf{x}^{*}_{j}$ are mapped to the discriminant subspace $\mathbf{z}^{*}_{j}=\mathbf{W}^{T}\mathbf{x}^{*}_{j}$ and $\mathbf{z}^{*}_{j},\>i=1,\dots,N$ are obtained. Then, $\mathbf{z}^{*}_{j}$’s are ordered based on their distance w.r.t. the positive class mean d_{j}=\|\mathbf{z}^{*}_{j}-\mbox{\boldmath\mu}\|_{2}, where $\mu$ is the mean of positive training samples in $\mathbb{R}^{d}$. The process followed in the test phase of PCSDA is illustrated in Algorithm 2.

III-C Spectral Regression solution of PCSDA

We further show in the following that PCSDA can be efficiently solved by following a spectral regression based process. Let us denote by $\mathbf{1}_{p}\in\mathbb{R}^{N}$ a vector having ones in the elements corresponding to the positive training vectors and zeros elsewhere. In the same manner, we define the vector $\mathbf{1}_{k}\in\mathbb{R}^{N}$ as a vector having ones in the elements corresponding to the negative samples belonging to the $k$-th subclass and zeros elsewhere. We also define by $\mathbf{J}_{p}\in\mathbb{R}^{N\times N}$ and $\mathbf{J}_{k}\in\mathbb{R}^{N\times N}$ the diagonal matrices having in their diagonal the vectors $\mathbf{1}_{p}$ and $\mathbf{1}_{k}$, respectively. Then, $\mathbf{S}_{n}$ and $\mathbf{S}_{I}$ can be expressed as:

[TABLE]

[TABLE]

where

[TABLE]

[TABLE]

Substituting (28) and (29) in (20) and setting $\mathbf{v}=\mathbf{X}^{T}\mathbf{w}$:

[TABLE]

Thus, the vectors $\mathbf{v}$ are the eigenvectors of the matrix $\mathbf{L}=\tilde{\mathbf{L}}^{-1}_{w}\mathbf{L}_{n}$, where $\tilde{\mathbf{L}}_{w}$ is a regularized version of $\mathbf{L}_{w}$, i.e. $\tilde{\mathbf{L}}_{w}=\mathbf{L}_{w}+\epsilon\mathbf{I}$ where $\epsilon>0$. It can be shown that the matrix $\mathbf{L}$ is a block matrix formed by blocks the elements of which are indicated by the class labels of the positive data and the cluster labels of the negative data. The top $K$ eigenvectors of it are also formed by blocks indicated by the same labels. Thus, they can be defined without the need of solving the problem (32), in a similar manner as in [19, 21]. After the determination of the matrix $\mathbf{V}=[\mathbf{v}_{1},\dots,\mathbf{v}_{K}]$, the solution of (20) is given by $\mathbf{W}=\left(\mathbf{X}^{T}\right)^{{\dagger}}\mathbf{V}$, where $\left(\mathbf{X}^{T}\right)^{{\dagger}}$ is the pseudo-inverse of $\mathbf{X}^{T}$.

III-D CSDA variants under the probabilistic model

A special case of the PCSDA can be obtained by setting the assumption that each negative sample forms a negative subclass, i.e. $K=N_{n}$ and $M=1$. In that case $\mathbf{\Phi}_{w}=\mathbf{0}$, $\mathbf{\Phi}_{O}=\mathbf{\Phi}_{n}$, the negative samples are drawn from a distribution $P(\mathbf{x})\sim N(\cdot|\mathbf{m},\mathbf{\Phi}_{n})$, and $\mathbf{W}$ is calculated by solving for $\mathbf{S}_{n}\mathbf{w}=\tilde{\lambda}\mathbf{S}_{p}\mathbf{w}$, i.e., we obtain the class discrimination definition of CSDA. The Spectral Regression-based solutions of CSDA in [1, 2, 20] and the low-rank regression solution of [21] use the same class discrimination criterion and, thus, correspond to the same setting of PCSDA. Since the discrimination criterion used in CSDA is a special case of the proposed probabilistic model, all the above-mentioned methods can be extended to perform classification using $g(\cdot)$ in (27). Jointly optimizing the data projection and the classification rule can be considered a big advantage.

III-E Class-specific vs. Contrastive Learning

As can be seen from Eqs. (1), (2) and (3), the Class-specific Discriminant Analysis criterion has similarities with Contrastive Learning based on triplet loss [23], which is defined for multi-class problems. That is, in its generic form, it optimizes for class discrimination by using three samples, i.e. the sample of interest, a sample belonging to the same class with the sample of interest (defining the same-label pair), and a sample belonging to a different class (defining the different-label pair). Thus, it can be modeled by a graph-based criterion, as was shown in [24]. The class-specific learning approaches are defined on binary problems where the negative class is formed data coming from multiple classes, but are assigned the same (negative) label. All the class-specific discriminant analysis approaches (including the proposed one) are defined using variance criteria. They optimize for class discrimination by using the mean of the positive class, a positive sample, and a negative sample. Thus, they can be defined as graph-based methods having the Laplacian matrices $\mathbf{L}_{n}$ and $\mathbf{L}_{I}$ in Eqs. (30) and (31).

The adoption of class-specific learning has two advantages compared to contrastive learning when class-specific subspace learning is considered:

• •

Subspace learning based on triplet loss optimizes for two objectives:

Given a positive vector of interest, it should be mapped close to another positive vector (same-label pair), and far away from a negative vector (different-label pair). 2. 2.

Given a negative vector of interest, it should be mapped close to another negative vector (same-label pair), and far away from a positive vector (different-label pair).

While objective $1$ is desirable in class-specific problems, optimizing for objective $2$ can be destructive. Optimizing for both objectives is good for multi-class problems when assuming class unimodality, like in the case of LDA.

• •

Class-specific learning defined by a variance-based criterion can lead to fast solutions. This is due to that the graph Laplacian matrices $\mathbf{L}_{n}$ and $\mathbf{L}_{I}$ do not need to be calculated and stored in the memory. Instead, their eigenvectors can be efficiently obtained and highly reduce the memory and computational cost. This is described in Section III-C.

III-F Non-linear PCSDA

In the above analysis we considered the linear class-specific subspace learning case. In order to non-linearly map $\mathbf{x}_{i}\in\mathbb{R}^{D}$ to $\mathbf{z}_{i}\in\mathbb{R}^{d}$ traditional kernel-based learning methods perform a non-linear mapping of the input space $\mathbb{R}^{D}$ to the feature space $\mathcal{F}$ using a function $\phi(\cdot)$, i.e. $\mathbf{x}_{i}\in\mathbb{R}^{D}\rightarrow\phi(\mathbf{x}_{i})\in\mathcal{F}$. Then, linear class-specific projections are defined by using the training data in $\mathcal{F}$. Since the dimensionality of $\mathcal{F}$ is arbitrary (virtually infinite), the data representations in $\mathcal{F}$ cannot be calculated. Traditional kernel-based learning methods address this issue by exploiting the Representer theorem and the non-linear mapping is implicitly performed using the kernel function encoding dot products in the feature space, i.e. $\kappa(\mathbf{x}_{i},\mathbf{x}_{j})=\phi(\mathbf{x}_{i})^{T}\phi(\mathbf{x}_{j})$ [25].

As has been shown in [26] the effective dimensionality of the kernel space $\mathcal{F}$ is at most equal to $L=\min(D,N)$ and, thus, an explicit non-linear mapping $\mathbf{x}_{i}\in\mathbb{R}^{D}\rightarrow\mathbf{\phi}_{i}\in\mathbb{R}^{L}$ can be calculated such that $\kappa(\mathbf{x}_{i},\mathbf{x}_{j})=\mathbf{\phi}_{i}^{T}\mathbf{\phi}_{j}$. This is achieved by using $\mathbf{\Phi}=\mathbf{\Sigma}^{\frac{1}{2}}\mathbf{U}^{T}$, where $\mathbf{U}$ and $\mathbf{\Sigma}$ contain the eigenvectors and eigenvavlues of the kernel matrix $\mathbf{K}\in\mathbb{R}^{N\times N}$ [27]. Thus, extension of PCSDA to the non-linear (kernel) case can be readily obtained by applying the above-described linear PCSDA on the vectors $\mathbf{\phi}_{i},\>i=1,\dots,N$. Here we need to mention that, since the proposed method exploits kernel-based data representations, it inherits drawbacks of kernel methods related to their high computational and memory costs when applied to large-scale problems. For the cases where the size of training set is prohibitive for applying kernel-based discriminant learning, the Nyström-based kernel subspace learning method of [28] or nonlinear data mappings based on randomized features, as proposed in [29], can be used. We should also note that the application of $K$-Means using in $\mathbb{R}^{L}$ corresponds to the application of kernel $K$-Means in the original space $\mathbb{R}^{D}$.

III-G Time complexity

The time complexity of PCSDA is as follows [30]:

• •

$K$-Means application for determining the subclasses of the negative class having time complexity of $O(\eta KN_{n}D)$, where $\eta$ is the number of iterations until convergence,

• •

Calculation of the scatter matrices in (20) and solution of the generalized eigen-analysis problem having a time complexity of $O(D^{3}+D^{2}N)$,

Keeping the high-order terms, the overall time complexity is of $O(D^{3}+D^{2}N)$.

Considering the kernel-based version of PCSDA, its time complexity is as follows:

• •

Kernel matrix $\mathbf{K}$ calculation having a time complexity of $O(DN^{2})$.

• •

Application of the method [26], involving the eigen-decomposition of $\mathbf{K}$, having a time complexity of $O(N^{3})$.

• •

$K$-Means application for determining the negative class subclasses having complexity of the order of $O(\eta KN_{n}N)$.

• •

Calculation of the scatter matrices in (20) and solution of the generalized eigen-analysis problem having a time complexity of $O(N^{3}+2N^{2})$

Thus, the time complexity of the kernel version of PCSDA is $O(N^{3})$, which is the case of a generic kernel-based subspace learning method. As noted in subsection III-F, for applying nonlinear PCSDA on big datasets, the approximate kernel subspace learning method in [28] or nonlinear data mappings based on randomized features, as proposed in [29], can be used leading to $L\ll N$ and highly reducing the overall time complexity to $O(L^{3}+LN^{2})$.

Finally, considering the Spectral Regression-based solution described in Section III-C, the time complexity of a generic solution is the same as those described above, but efficient solutions are possible by using efficient decomposition methods, like Cholesky decomposition as in [21].

IV Experiments

In this Section we provide experimental results illustrating the performance of the proposed PCSDA method. We used eight datasets in our experiments, details of which are illustrated in Table I. For BU, Jaffe, Kanade, YALE and AR facial image datasets we used the vectorized pixel intensity values for representing the images. For the $15$ scenes and Caltech101 datasets we used deep features generated by average pooling over spatial dimension of the last convolution layer of VGG network [39] trained on ILSVRC2012 database. For optDigits dataset we used the raw features provided by the UCI repository [37].

On each experiment, we formed class-specific ranking and classification problems for each class using the class data as positive samples and the data of the remaining classes as negative samples. In all the experiments the data is non-linearly mapped to the subspace of the kernel space (as discussed in Section III-F). We used the RBF kernel function and set the value of $\sigma$ equal to the mean pair-wise distance value between the positive training vectors. For the small datasets, we used the method in [27] by keeping the eigenvectors corresponding to all positive eigenvalues, while for the large datasets we used the method in [28] by setting the dimensionality of the resulted kernel subspace to $L=1000$. On each class-specific problem, we ran five experiments by randomly selecting $70\%$ of the positive and negative classes for training and the rest $30\%$ for testing and measure the average performance value. The hyper-parameter values of all methods have been optimized by applying five-fold cross-validation on the training data. Specifically, for the methods performing subspace learning, the dimensionality of the obtained subspace was selected within the range of $[1,25]$ and for Support Vector Machine (SVM), Support Vector Regression (SVR) and Ridge Regression (RR) the regularization parameter value was selected in the range of $10^{r},\>r=-3,\dots,3$. We used the publicly available software implementations from [40] for the support vector methods. We tested two variants of the proposed PCSDA, one in which the number of negative subclasses is fixed to $K=1$ (noted as PCSDA-1) and one in which the number of negative subclasses is automatically selected during the training process following the five-fold cross validation process from the set $K=\{5,10,15,20\}$ (noted as PCSDA-K). We opt to automatically determine the number of negative subclasses (and not set $K$ equal to the number of the classes of the multi-class classification problem forming the negative class) to appropriately address possible issues of class multi-modality. Indeed, by following such an approach, negative subclasses can be formed by samples belonging to more than one classes of the original multi-class classification problem, and multiple negative subclasses can be formed by samples belonging to the same class of the original multi-class classification problem. We should note here that the case $K=1$ is a subcase of the generic PCSDA-K.

For evaluating the performance of the methods in a ranking setting, we use the mean Average Precision (mAP) metric. For evaluating the performance of the methods in a classification setting, we use the $f_{1}$ score defined as $f_{1}=2\>\frac{precision*recall}{precision+recall}$. Performance obtained for each method in a ranking and a classification setting is shown in Tables II and III, respectively. The values of the Tables correspond to the mean performance value and the corresponding standard deviation over all class-specific problems of each dataset.

As can be seen in Table II for the ranking problems, the use of an unsupervised subspace learning method (PCA) leads to low performance. The use of regression models (RR and SVR) achieves high performance in most of the datasets, except Kanade and BU. The case is similar for the supervised subspace learning methods (LDA and PCSDA). It is interesting to see that in these two datasets PCSDA outperforms LDA by a margin of around $3\%$, while for the rest datasets their performance is similar. Focusing on PCSDA-1 and PCSDA-K, we can see that in most cases the use of one negative subclass seems to provide (slightly) better results, except for the Yale and Caltech101 datasets. We speculate that this is due to the homogeneity of the remaining datasets, however, we can see that the use of multiple subclasses can enhance performance for heterogeneous data.

Table III illustrates the performance of the methods in classification problems. Here we need to note that for performing classification using a subspace-based method (PCA and LDA) one needs to perform a two-step process, i.e. data projection to the subspace followed determined by the method followed by the application of a classification method. We combine PCA with SVM and LDA with Nearest Class Centroid (NCC) classifiers. The choice of LDA+NCC is based on the fact that the LDA optimization problem exploits as a class separability criterion that of NCC. Despite the fact that PCSDA is a subspace learning method, its probabilistic formulation leads to a classification rule (as detailed in Section III-B), and thus there is no need to perform a classification method for PCSDA. We can again see that the use of unsupervised subspace learning combined with SVM leads to low performance. Subspace learning-based classification leads to good performance, outperforming regression-based classification and SVM in most cases. Interestingly, the use of multiple negative subclasses leads to an improvement on Caltech101, Yale and OptDigits datasets of sizes $0.5\%$, $2\%$ and $2.5\%$, respectively.

V Conclusions

In this paper we proposed a probabilistic model for class-specific discriminant subspace learning that is able to incorporate subclass information of the negative class in the optimization criterion. The adoption of a probability-based optimization criterion for class-specific discriminant subspace learning leads to a classifier defined on the data representations in the discriminant subspace. We showed that the parameters of the probabilistic model can be obtained by applying an efficient Spectral Regression-based process. Moreover, we showed that the proposed probabilistic model includes as special cases existing class-specific discriminant analysis methods. Experimental results illustrated the performance of the proposed model, in comparison with that of related methods, in both verification and classification problems. Interesting extensions of the proposed approach would be to devise class-specific criteria for Subspace Clustering [41, 42] and multi-view based subspace learning [43, 44].

Acknowledgement

This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871449 (OpenDR). This publication reflects the authors’ views only. The European Commission is not responsible for any use that may be made of the information it contains.