Convex Formulation of Multiple Instance Learning from Positive and Unlabeled Bags

TL;DR

This paper introduces a convex positive-unlabeled (PU) learning approach for multiple instance learning (MIL), enabling effective learning from positive and unlabeled bags with improved performance and lower computational costs.

Contribution

It presents a novel convex PU learning framework specifically designed for MIL, addressing the challenge of limited labeled data in practical scenarios.

Findings

Achieves better performance than existing PU-MIL methods

Significantly reduces computational costs

Effective in scenarios with limited labeled bags

Abstract

Multiple instance learning (MIL) is a variation of traditional supervised learning problems where data (referred to as bags) are composed of sub-elements (referred to as instances) and only bag labels are available. MIL has a variety of applications such as content-based image retrieval, text categorization and medical diagnosis. Most of the previous work for MIL assume that the training bags are fully labeled. However, it is often difficult to obtain an enough number of labeled bags in practical situations, while many unlabeled bags are available. A learning framework called PU learning (positive and unlabeled learning) can address this problem. In this paper, we propose a convex PU learning method to solve an MIL problem. We experimentally show that the proposed method achieves better performance with significantly lower computational costs than an existing method for PU-MIL.

Tables

Table 1. Table 1: Comparison of existing and proposed discriminative methods for MIL.

	Convex	Non-convex
Positive and Negative	set kernels [20] sMIL [24] KI-SVM [22] miGraph [25]	MI-SVM [15] MissSVM [26] soft-bag SVM [10] dMIL [11]
Positive and Unlabeled	PU-SKC (Sect. 3)	puMIL [27]

Table 2. Table 2: Details of datasets: The last two rows indicate the numbers of instances per bag, which are the average with standard deviation.

Number of	Musk1	Musk2	Elephant	Fox	Tiger
features	166	166	230	230	230
positive bags	47	39	100	100	100
negative bags	45	63	100	100	100
positive instances	2.3 (2.6)	10.0 (26.1)	3.8 (4.2)	3.2 (3.6)	2.7 (3.1)
negative instances	2.9 (6.9)	54.7 (176.0)	3.2 (3.6)	3.4 (3.8)	3.4 (3.8)

Table 3. Table 3: Each result is the average with standard deviation of the classification accuracy over 20 trials. Bold faces represent the best methods under each setting (tested by the one-sided 5% t-test).

dataset	$\pi$	PU-SKC	LSDD [34]	DSDD [33]	puMIL [27]
Musk1	0.1	0.865 (0.046)	0.928 (0.029)	0.931 (0.026)	0.757 (0.065)
	0.2	0.844 (0.038)	0.707 (0.289)	0.876 (0.037)	0.733 (0.070)
	0.3	0.818 (0.041)	0.622 (0.208)	0.778 (0.045)	0.717 (0.063)
	0.4	0.776 (0.057)	0.618 (0.148)	0.708 (0.041)	0.699 (0.050)
	0.5	0.763 (0.050)	0.553 (0.072)	0.597 (0.047)	0.665 (0.081)
	0.6	0.735 (0.055)	0.522 (0.068)	0.505 (0.051)	0.649 (0.069)
	0.7	0.737 (0.044)	0.538 (0.161)	0.392 (0.061)	0.606 (0.075)
Musk2	0.1	0.810 (0.060)	0.702 (0.303)	0.840 (0.040)	0.688 (0.098)
	0.2	0.802 (0.053)	0.691 (0.239)	0.789 (0.036)	0.730 (0.063)
	0.3	0.801 (0.051)	0.621 (0.173)	0.720 (0.050)	0.732 (0.073)
	0.4	0.724 (0.063)	0.590 (0.111)	0.621 (0.048)	0.704 (0.068)
	0.5	0.742 (0.050)	0.522 (0.054)	0.537 (0.055)	0.654 (0.092)
	0.6	0.706 (0.059)	0.499 (0.086)	0.466 (0.053)	0.637 (0.086)
	0.7	0.726 (0.055)	0.508 (0.159)	0.364 (0.063)	0.599 (0.072)
Elephant	0.1	0.845 (0.044)	0.734 (0.153)	0.747 (0.041)	0.722 (0.084)
	0.2	0.783 (0.062)	0.671 (0.166)	0.715 (0.039)	0.686 (0.075)
	0.3	0.746 (0.062)	0.652 (0.092)	0.685 (0.038)	0.698 (0.072)
	0.4	0.701 (0.050)	0.573 (0.088)	0.608 (0.039)	0.642 (0.051)
	0.5	0.607 (0.058)	0.552 (0.052)	0.575 (0.050)	0.667 (0.070)
	0.6	0.528 (0.087)	0.520 (0.062)	0.503 (0.031)	0.614 (0.063)
	0.7	0.421 (0.059)	0.602 (0.141)	0.453 (0.037)	0.597 (0.066)
Fox	0.1	0.840 (0.053)	0.634 (0.181)	0.717 (0.037)	0.561 (0.062)
	0.2	0.754 (0.048)	0.689 (0.042)	0.669 (0.041)	0.575 (0.058)
	0.3	0.689 (0.054)	0.576 (0.130)	0.615 (0.045)	0.544 (0.045)
	0.4	0.613 (0.061)	0.542 (0.097)	0.588 (0.041)	0.552 (0.053)
	0.5	0.538 (0.042)	0.527 (0.046)	0.545 (0.042)	0.547 (0.074)
	0.6	0.468 (0.057)	0.538 (0.071)	0.464 (0.053)	0.550 (0.055)
	0.7	0.430 (0.055)	0.559 (0.151)	0.430 (0.046)	0.536 (0.075)
Tiger	0.1	0.839 (0.044)	0.689 (0.136)	0.730 (0.047)	0.706 (0.073)
	0.2	0.783 (0.035)	0.695 (0.032)	0.707 (0.031)	0.680 (0.070)
	0.3	0.729 (0.043)	0.636 (0.047)	0.651 (0.052)	0.659 (0.056)
	0.4	0.672 (0.043)	0.596 (0.042)	0.606 (0.036)	0.668 (0.048)
	0.5	0.584 (0.043)	0.566 (0.044)	0.540 (0.046)	0.609 (0.070)
	0.6	0.508 (0.048)	0.530 (0.066)	0.523 (0.032)	0.608 (0.080)
	0.7	0.410 (0.059)	0.490 (0.080)	0.463 (0.034)	0.585 (0.086)

Equations

$$f(\mathbf{x})$$

$$g$$

$$g(\mathbf{x})={\mathbf{\alpha }}^{\top }\mathbf{\varphi }(\mathbf{x})+\beta ,$$

$$\underset{\mathbf{\alpha }\in {\mathbb{R}}^m,\beta \in \mathbb{R}}{\min }$$

$$f(X)$$

$$g$$

$$\tilde{k}(X,X^{\prime }):=k(\mathbf{s}(X),\mathbf{s}(X^{\prime })),$$

$${\mathbf{s}}_{\mathrm{minimax}}(X)$$

$${\tilde{k}}_{\mathrm{minimax}}(X,X^{\prime })$$

$$g(X)$$

$$\mathbf{\varphi }(X)$$

$$\{{\mathbf{x}}_i^{\mathrm{P}}{\}}_{i=1}^{N_{\mathrm{P}}}\stackrel{i.i.d.}{\sim }$$

$$\{{\mathbf{x}}_i^{\mathrm{U}}{\}}_{i=1}^{N_{\mathrm{U}}}\stackrel{i.i.d.}{\sim }$$

$$=\pi p_{+}(\mathbf{x})+(1-\pi )p_{-}(\mathbf{x}),$$

$$R_{0\text{mbox}-1}(g)=\pi {\mathbb{E}}_{\mathrm{P}}\left[{\ell }_{0\text{mbox}-1}(g(\mathbf{x}))\right]+(1-\pi ){\mathbb{E}}_{\mathrm{N}}\left[{\ell }_{0\text{mbox}-1}(-g(\mathbf{x}))\right],$$

$$\displaystyle\ell_{0\mbox{-}1}(z)=\begin{cases}0&\mbox{if $z\geq 0$,}\\ 1&\mbox{otherwise.}\end{cases}$$

$$R(g):=\pi {\mathbb{E}}_{\mathrm{P}}\left[\ell (g(\mathbf{x}))\right]+(1-\pi ){\mathbb{E}}_{\mathrm{N}}\left[\ell (-g(\mathbf{x}))\right].$$

$$(1-\pi ){\mathbb{E}}_{\mathrm{N}}\left[\ell (-g(\mathbf{x}))\right]={\mathbb{E}}_{\mathrm{U}}\left[\ell (-g(\mathbf{x}))\right]-\pi {\mathbb{E}}_{\mathrm{P}}\left[\ell (-g(\mathbf{x}))\right].$$

$$R(g)=\pi {\mathbb{E}}_{\mathrm{P}}\left[\ell (g(\mathbf{x}))-\ell (-g(\mathbf{x}))\right]+{\mathbb{E}}_{\mathrm{U}}\left[\ell (-g(\mathbf{x}))\right],$$

$$\ell (z)-\ell (-z)=-z,$$

$$R(g)=\pi {\mathbb{E}}_{\mathrm{P}}\left[-g(\mathbf{x})\right]+{\mathbb{E}}_{\mathrm{U}}\left[\ell (-g(\mathbf{x}))\right].$$

$${\ell }_{\mathrm{S}}(z)$$

$${\ell }_{\mathrm{LL}}(z)$$

$${\ell }_{\mathrm{DH}}(z)$$

$$J(g)=\pi {\mathbb{E}}_{\mathrm{P}}\left[-g(X)\right]+{\mathbb{E}}_{\mathrm{U}}\left[\ell (-g(X))\right].$$

$$g(X)={\mathbf{\alpha }}^{\top }\mathbf{\varphi }(X)+\beta ,$$

$$\displaystyle\bm{\phi}(X)=\left[\begin{array}[]{c}\widetilde{k}_{\mathrm{minimax}}(X,X^{\mathrm{P}}_{1})\\ \vdots\\ \widetilde{k}_{\mathrm{minimax}}(X,X^{\mathrm{P}}_{N_{\mathrm{P}}})\\ \widetilde{k}_{\mathrm{minimax}}(X,X^{\mathrm{U}}_{1})\\ \vdots\\ \widetilde{k}_{\mathrm{minimax}}(X,X^{\mathrm{U}}_{N_{\mathrm{U}}})\end{array}\right].$$

$$\hat{J}(\mathbf{\alpha },b)$$

$$+\frac{1}{N_{\mathrm{U}}}\sum _{b^{\prime }=1}^{N_{\mathrm{U}}}{\ell }_{\mathrm{DH}}\left(-{\mathbf{\alpha }}^{\top }\mathbf{\varphi }(X_{b^{\prime }}^{\mathrm{U}})-\beta \right)+\frac{\lambda }{2}{\mathbf{\alpha }}^{\top }\mathbf{\alpha },$$

$$\underset{\mathbf{\alpha },\beta ,\mathbf{\xi }}{\min }$$

$$\mathbf{\xi }\ge 0,$$

$$\mathbf{\xi }\ge \frac{1}{2}1+\frac{1}{2}{\Phi }_{\mathrm{U}}\mathbf{\alpha }+\frac{1}{2}\beta 1,$$

$$\mathbf{\xi }\ge {\Phi }_{\mathrm{U}}\mathbf{\alpha }+\beta 1,$$

$$\Phi_{\mathrm{P}}=\left[\begin{array}[]{c}\bm{\phi}^{\top}(X^{\mathrm{P}}_{1})\\ \vdots\\ \bm{\phi}^{\top}(X^{\mathrm{P}}_{N_{\mathrm{P}}})\end{array}\right],\quad\Phi_{\mathrm{U}}=\left[\begin{array}[]{c}\bm{\phi}^{\top}(X^{\mathrm{U}}_{1})\\ \vdots\\ \bm{\phi}^{\top}(X^{\mathrm{U}}_{N_{\mathrm{U}}})\end{array}\right].$$

$$\pi ={\left[2{\hat{\mathbf{h}}}^{\top }{\hat{G}}^{-1}\hat{\mathbf{h}}-{\hat{\mathbf{h}}}^{\top }{\hat{G}}^{-1}\hat{H}{\hat{G}}^{-1}\hat{\mathbf{h}}\right]}^{-1},$$

$$\hat{H}=\frac{1}{N_{\mathrm{P}}}\sum _{b=1}^{N_{\mathrm{P}}}\mathbf{\varphi }(X_b^{\mathrm{P}})\mathbf{\varphi }(X_b^{\mathrm{P}}{)}^{\top },\hat{\mathbf{h}}=\frac{1}{N_{\mathrm{U}}}\sum _{b^{\prime }=1}^{N_{\mathrm{U}}}\mathbf{\varphi }(X_{b^{\prime }}^{\mathrm{U}}),\hat{G}=\hat{H}+\eta I,$$

Full text

1 Introduction

Multiple instance learning (MIL) [1] is a learning problem with bags and instances. Instances are the same as ordinary feature vectors, while bags are sets of instances. The numbers of instances in different bags varies. Bag labels are defined as follows.

If a bag contains at least one positive instance, then its label is positive.

• 2.

If a bag contains no positive instances, then its label is negative.

This is the basic setup of MIL. The goal of MIL is to predict labels of test bags. MIL is more difficult than ordinary classification problems because instance labels are unavailable.

MIL was originated from molecule/graph data [1], where ray-based representation is used to describe molecule shapes. Later, MIL has been considered as a graph-based learning problem [2, 3, 4, 5, 6]. In fact, MIL is applicable to a wide range of real-world problems such as molecule behavior prediction [7], drug activity prediction [1], domain theory [8], content-based image retrieval [9, 10, 11], visual tracking [12], object detection [13, 14], text categorization [15], and medical diagnosis [16, 17].

So far, a lot of approaches for MIL have been developed [1, 18, 19, 15, 20, 21], which are classified into two groups in general.

Methods in the first group are based on generative modeling, including the diverse density [18] and its extension, the expectation-maximization diverse density (EM-DD) [19]. These methods find out an instance close to instances in training positive bags and far from instances in training negative bags, which is referred to as a concept point. This process is carried out by gradient-based search from every training instance, which is computationally inefficient. 2. 2.

Methods in the second group are based on discriminative modeling. The multiple-instance support vector machine (MI-SVM) [15] is an approach based on SVMs. Empirical evaluation shows that MI-SVM performs well, but its optimization problem is non-convex and finding a solution is computationally expensive. The key-instance support vector machine (KI-SVM) [22] reformulates the optimization problem of MI-SVM as mixed-integer programming, which is still hard to optimize. Gärtner et al. [20] introduced set kernels (a.k.a. multiple instance kernels), which are extensions of the standard kernel functions to MIL. The set kernels can be used to construct a standard SVM classifier, which performs well in experiments. The optimization problem in this training procedure is convex and the global solution can be obtained efficiently.

In this work, we propose a novel method to construct multiple instance classifiers only from positive and unlabeled bags, while the above standard approaches to MIL assume that training bags are fully labeled. This problem is called PU-MIL. For example, PU-MIL is applicable to the following situations.

The situation where it is difficult to obtain an enough amount of labeled data due to the significant labeling costs, such as outlier detection based on supervised classification, where it is often difficult to label all outlier samples. On the other hand, in PU-MIL, we need to label only some of outlier samples and the rest can be regarded as unlabeled.

• 2.

The situation where the true negative labels are essentially unavailable, such as bioinformatics and cheminformatics. MIL setting commonly appears in these natural science fields [1]. In natural science, experiments are often designed to observe some phenomena (detect positives), not designed to deny the existence of the phenomena. Thus even if we did not observe the phenomenon, it might not be appropriate to say “the phenomenon did not occur.” In other words, there might be false negatives. PU setting plays an important role in this kind of situations.

Our contribution in this paper is to propose a novel PU-MIL method based on empirical risk minimization [23]. The proposed method formulates an optimization problem as a convex optimization problem together with a linear-in-parameter model, and the global optimal solution can be computed efficiently. To the best of our knowledge, this is the first convex PU-MIL method (see Table 1). Through experiments, we demonstrate that the proposed method combined with the minimax kernel [15] compares favorably with an existing method.

The rest of this paper is structured as follows. In Sect. 2, we review existing methods for PU classification [28, 23] and MIL [20], on which our proposed method is based. In Sect. 3, we explain the formulation and optimization algorithm of our proposed method, called the positive and unlabeled set kernel classifier (PU-SKC). In Sect. 5, we experimentally compare the performance of the proposed method (PU-SKC) with an existing method (puMIL) [27]. Finally, we conclude this work in Sect. 6.

2 Problem Formulation and Related Work

In this section, we formulate the problems we discuss in this paper (see Fig. 1) and review related work.

2.1 Ordinary Binary Classification

Let $\bm{x}\in\mathbb{R}^{d}$ be a $d$-dimensional feature vector and $y\in\{+1,-1\}$ be its corresponding class label. In the ordinary binary classification problem, we construct a binary classifier

[TABLE]

from an i.i.d. training dataset $\mathcal{D}=\{(\bm{x}_{i},y_{i})\}_{i=1}^{N}$, where $N$ is the number of training samples. Here we use a linear-in-parameter model for $g$:

[TABLE]

where $\top$ denotes the transpose, $\bm{\alpha}\in\mathbb{R}^{m}$ is an $m$-dimensional parameter vector, $\beta\in\mathbb{R}$ is a bias parameter, and $\bm{\phi}:\mathbb{R}^{d}\to\mathbb{R}^{m}$ is a vector of basis functions. The support vector machine (SVM) [29] is one of the most standard methods for training a binary classifier. The optimization problem of SVM is given as follows:

[TABLE]

where $C>0$ is a penalty parameter. This problem can be reformulated as a quadratic program (QP), which can be solved efficiently.

2.2 Multiple Instance Learning

We formulate the problem of multiple instance learning (MIL) and review an existing method.

2.2.1 Formulation

Hereafter $\mathcal{P}(A)$ denotes the power set111 The power set of $A$ is a set of all subsets of $A$, including $\emptyset$ and $A$ itself. In the MIL setting, bags belong to $\mathcal{P}(\mathbb{R}^{d})$, i.e., bags are composed of some elements in $\mathbb{R}^{d}$. of $A$. Let $X=\{\bm{x}_{i}|\bm{x}_{i}\in\mathbb{R}^{d}\}_{i=1}^{n}\in\mathcal{P}(\mathbb{R}^{d})\setminus\emptyset$ be a bag containing $n$ instances whose dimensions are $d$, and $Y\in\{+1,-1\}$ be a bag label corresponding to $X$. The problem is to construct a binary classifier:

[TABLE]

from an i.i.d. fully-labeled training dataset $\mathcal{D}=\{(X_{b},Y_{b})\}_{b=1}^{N}$, where $N$ denotes the number of bags in $\mathcal{D}$.

2.2.2 Multiple Instance Kernels

Gärtner et al. [20] proposed set kernels (multiple instance kernels), which map bags (sets of instances) to a feature space. A type of the set kernels, called the statistic kernel $\widetilde{k}$, is defined as follows:

[TABLE]

where $k$ is an arbitrary kernel function such as the Gaussian kernel, and $\bm{s}$ is called a statistic. For example, the following minimax statistics is a typical choice:

[TABLE]

where $x^{(i)}$ is the $i$-th element of an instance $\bm{x}$ in the bag $X$. Gärtner et al. [20] experimentally demonstrated that the statistic kernel with the minimax statistics (4) for $\bm{s}$ and the polynomial kernel for $k$ shows good performance:

[TABLE]

where $\rho$ is a positive integer. The statistic kernel (5) is referred to as the minimax kernel. We can then construct the following set kernel classifier $g$:

[TABLE]

where $C_{1},\ldots,C_{M}$ are kernel centers and $M$ is the number of kernel centers. We can obtain the MIL classifier by using SVM (2) to train the classifier (6).

2.3 Learning from Positive and Unlabeled Data

We formulate a binary classification problem from positive and unlabeled instances and review existing methods.

2.3.1 Formulation

We assume that positive samples $\{\bm{x}_{i}^{\mathrm{P}}\}_{i=1}^{N_{\mathrm{P}}}$ and unlabeled samples $\{\bm{x}_{i}^{\mathrm{U}}\}_{i=1}^{N_{\mathrm{U}}}$ are generated as follows:

[TABLE]

where $\pi:=p(y=+1)$ is called the class prior. Our objective is to construct the binary classifier (1) only from positive and unlabeled samples.

2.3.2 Learning Instance-Level Classifiers from Positive and Unlabeled Data

du Plessis et al. [28, 23] proposed methods based on empirical risk minimization to learn only from positive and unlabeled samples. In the ordinary binary classification setting, an optimal classifier $g^{*}$ minimizes the following misclassification rate:

[TABLE]

where $\mathbb{E}_{{\mathrm{P}}}\left[{\cdot}\right]$ and $\mathbb{E}_{{\mathrm{N}}}\left[{\cdot}\right]$ denote the expectations over $p_{+}(\bm{x})$ and $p_{-}(\bm{x})$ respectively and $\ell_{0\mbox{-}1}$ denotes the zero-one loss:

[TABLE]

In practice, the misclassification rate (12) is difficult to optimize because the subgradient of $\ell_{0\mbox{-}1}$ is always [math] except at $z=0$. For this reason, we usually use a surrogate loss function222 For example, the hinge loss $\ell_{\mathrm{H}}(z)=\max(-z,0)$ and the ramp loss $\ell_{\mathrm{R}}(z)=\frac{1}{2}\max(0,\min(2,1-z))$ are commonly used [30]. . Then the risk function $R$ with the surrogate loss function $\ell$ is written as

[TABLE]

Since negative samples are not available in the PU classification setup, let us consider expressing the risk (13) without $\mathbb{E}_{{\mathrm{N}}}\left[{\cdot}\right]$. By the definition of the unlabeled sample distribution (11), the following equation holds:

[TABLE]

Substituting this into the risk (13), we obtain

[TABLE]

where $\mathbb{E}_{{\mathrm{U}}}\left[{\cdot}\right]$ denotes the expectation over $p(\bm{x})$. If the surrogate loss function $\ell$ satisfies

[TABLE]

the risk (14) can be written as

[TABLE]

The risk (16) is convex if the surrogate loss function $\ell$ is convex. Convex loss functions such as the squared loss $\ell_{\mathrm{S}}$, the logistic loss $\ell_{\mathrm{LL}}$, and the double hinge loss $\ell_{\mathrm{DH}}$ satisfy the condition (15):

[TABLE]

We use the risk (16) to obtain a convex formulation of PU-MIL in Sect. 3.

3 Positive and Unlabeled Set Kernel Classifier

In this section, we propose a convex method for PU-MIL, named the PU-SKC (positive and unlabeled set kernel classifier).

3.1 Multiple Instance Learning from Positive and Unlabeled Bags

We formulate the problem of multiple instance learning from positive and unlabeled bags (PU-MIL). The purpose of PU-MIL is to construct the bag-level classifier (3) from a positively labeled training dataset $\mathcal{D}_{\mathrm{P}}=\{(X_{b}^{\mathrm{P}},Y_{b}=+1)\}_{b=1}^{N_{\mathrm{P}}}$ and an unlabeled training dataset $\mathcal{D}_{\mathrm{U}}=\{X_{b^{\prime}}^{\mathrm{U}}\}_{{b^{\prime}}=1}^{N_{\mathrm{U}}}$, where $N_{\mathrm{P}}$ and $N_{\mathrm{U}}$ denote the number of positive bags in $\mathcal{D}_{\mathrm{P}}$ and the number of unlabeled bags in $\mathcal{D}_{\mathrm{U}}$, respectively. We assume that $X_{1}^{\mathrm{P}},\dots,X_{N_{\mathrm{P}}}^{\mathrm{P}}\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}p(X|Y=+1)$ and $X_{1}^{\mathrm{U}},\dots,X_{N_{\mathrm{U}}}^{\mathrm{U}}\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}p(X)$.

3.2 Formulation

As we mentioned in Sect. 2.3, du Plessis et al. [28, 23] formulated the PU classification problem in the empirical risk minimization framework. If we use a loss function $l(z)$ such that $l(z)-l(-z)=-z$, we have the following objective function:

[TABLE]

Here we use a linear-in-parameter model with the set kernel function as a classifier:

[TABLE]

where $\bm{\phi}$ is a vector of basis functions:

[TABLE]

As with the standard binary classification, we predict a given bag as positive if $g(X)\geq 0$, and as negative if $g(X)<0$.

The risk (18) together with the bag-level classifier (19) and the $l_{2}$ regularizer induces the following objective function to be minimized:

[TABLE]

where $\lambda\geq 0$ is the regularization parameter. Here we use the double hinge loss $\ell_{\mathrm{DH}}$ (17) because du Plessis et al. [23] reported that it achieved the best performance in the ordinary PU classification setting. Note that $\pi$ is the bag-level class prior, i.e., $\pi=p(Y=+1)$, which must be estimated from the training data. We explain how to estimate it in Sect. 3.3.

The problem of minimizing (27) can be rewritten in the form of a quadratic program by using slack variables $\bm{\xi}$ as

[TABLE]

where $\geq$ for vectors denotes the element-wise inequality, and $\bm{0}$, $\bm{1}$ denote the all-zero and all-one vectors, respectively. Matrices $\Phi_{\mathrm{P}}$ and $\Phi_{\mathrm{U}}$ are defined as follows:

[TABLE]

Note that our proposed method is independent of kernel choices. Other set kernels, such as the conformal kernels [31] and affinity propagation clustering-based feature representation [32], can also be used instead of the polynomial minimax kernel, depending on domain-specific knowledge.

3.3 Bag-Level Class Prior Estimation

A bag-level class prior estimation algorithm can be obtained by a simple extension of the instance-level version explained in A. The difference is basis functions used for estimating the class prior. We use the polynomial minimax kernel (5) to obtain the bag-level basis functions (26). Then the bag-level class prior $\pi$ can be estimated similarly:

[TABLE]

where

[TABLE]

and $\eta>0$ is another regularization parameter. The detailed derivation is described in A.

3.4 Remarks: Instance-Level PU-MIL

PU-SKC is a bag-level method, namely, classifying bags directly (Eq. (19)) instead of aggregating instance-level classification results. On the other hand, we can also consider an instance-level method to solve PU-MIL.

Assume that instances in negative bags are drawn from the instance-level negative conditional distribution, i.e.,

[TABLE]

and instances in positive bags are drawn from the instance-level marginal distribution, i.e.,

[TABLE]

where $\theta:=p(y=+1)$ is the instance-level class prior. Since we assume the bag-level class prior $p(Y=+1)=\pi$, instances in unlabeled bags are drawn from the following distribution, i.e.,

[TABLE]

From the instance-level perspective, both positive and unlabeled bags are unlabeled datasets, but the class proportions are different ($\theta$ for Eq. (30) and $\pi\theta$ for Eq. (31)). In fact, an (instance-level) binary classifier can be obtained from two distinct datasets with different class proportions [33].

Assume that the test class priors are equal $q(y=+1)=q(y=-1)=1/2$ and the test conditional density $q(\bm{x}|y)$ is equal to $p(\bm{x}|y)$. We begin with the difference of the class posteriors:

[TABLE]

Since $2q(\bm{x})$ is always positive, the classification criterion on the test distribution becomes

[TABLE]

On the other hand,

[TABLE]

Since $\theta(1-\pi)>0$, the classification criterion becomes333 In the original paper [33], an unknown constant $C=\pm 1$ is multiplied in the right-hand side. On the other hand, in our current setting, we know that the class priors are $\theta$ and $\pi\theta$ and this allows us to determine the sign of $C$.

[TABLE]

The point is, in order to obtain an instance-level classifier, all we have to do is to estimate the density difference $p(\bm{x})-p^{\prime}(\bm{x})$. To this end, a method called least-squares density difference (LSDD) estimation has been proposed [34]. A more advanced method to estimate the sign of the density difference $\mathrm{sign}[p(\bm{x})-p^{\prime}(\bm{x})]$ directly has also been proposed [33], which is called direct sign density difference (DSDD) estimation. These estimators are also compared as baselines in experiments.

The instance-level approach is useful when our goal is to determine all instance-level labels. Otherwise, the bag-level approach is suitable because it is a direct approach to determine only bag-level labels, which is referred to as Vapnik’s principle [35]:

When solving a problem of interest, one should not solve a more general problem as an intermediate step.

Here, knowing instance-level labels allows us to identify bag-level labels, but not vice versa. In this sense, obtaining an instance-level classifier is regarded as solving an intermediate/general problem when our goal is to predict labels for bags.

4 Analysis of Generalization Error Bounds

In this section, we show an upper bound of the generalization error (evaluated on a fixed classifier) for our proposed method. Let $\mathscr{X}:=\mathcal{P}(\mathbb{R}^{d})\setminus\emptyset$ be the bag-level domain set and

[TABLE]

be a given function class, where $\bm{\phi}$ is a vector of basis functions defined in Eq. (26). Note that $\mathcal{G}$ includes the classifier (19) as a special case444Let $\widetilde{\bm{\alpha}}:=[\bm{\alpha}\;\beta]^{\top}$ and $\widetilde{\bm{\phi}}(X):=[\bm{\phi}(X)\;1]^{\top}$ then it is included in $\mathcal{G}$.. Throughout this section, let $\ell:\mathbb{R}\rightarrow\mathbb{R}_{+}$ be the double hinge loss (17), which is used in our experiments. We denote the expected risk of a bag-level classifier $g:\mathscr{X}\rightarrow\mathbb{R}$ with respect to $\ell$ as

[TABLE]

and the corresponding empirical risk as

[TABLE]

where $\pi^{*}:=p(Y=+1)$ is the true class prior of the positive class.

Theorem 1**.**

For a fixed $g\in\mathcal{G}$, and for any $\delta\in(0,1)$, with probability at least $1-\delta$,

[TABLE]

where $C_{\mathcal{G},\delta}>0$ is a constant depending jointly on $\mathcal{G}$ and $\delta$.

The proof is in C. This theorem shows that the generalization error decreases with order $1/\sqrt{N_{\mathrm{P}}}$ and $1/\sqrt{N_{\mathrm{U}}}$. Thus, increasing the number of positive bags and the number of unlabeled bags both contributes to reducing the error. Note that this order is optimal in a parametric setup [30]. Furthermore, we can see from Eq. (34) that the true class prior $\pi^{*}$ and $N_{\mathrm{P}}$ are related in the generalization error bound, while $\pi^{*}$ and $N_{\mathrm{U}}$ are not. We will further investigate this issue through experiments in Section 5.

5 Experiments

In this section, we experimentally compare the proposed method555 Implementation is published at https://github.com/levelfour/pumil.

with the baselines (Sect. 3.4), and an existing method (see B) and give answers to the following research questions.

Q1: Does the proposed method outperform the baseline and existing methods regardless of the true class prior?

Q2: Is the proposed method computationally efficient?

5.1 Datasets

We used standard MIL datasets: Musk and Corel666http://www.cs.columbia.edu/~andrews/mil/datasets.html. The details of these benchmark datasets are shown in Table 2.

Since these datasets are too small to evaluate PU methods, we augmented them to increase the number of bags. Specifically, bags chosen randomly from the original datasets were duplicated and then Gaussian noise with mean zero and variance $0.01$ was added to each dimension. In this way, we increased the number of samples in the Musk datasets (Musk1 and Musk2) 10 times and the Corel datasets (Elephant, Fox, and Tiger) 5 times. After this augmentation process, we generated a training set (including labeled positive bags and unlabeled bags) and a test set. This generation process is described in Algorithm 1 (we set $L=20,U=180$, and $T=200$).

Remark: This dataset processing is needed. The reasons are as follows.

We assume that the training distribution and test distribution are same, which means that the class priors are same, too. Thus Algorithm 1 is needed to maintain the class priors to be same among both (unlabeled) training and test datasets. 2. 2.

If Algorithm 1 is applied, it is hard to obtain an enough number of negative bag samples under extremely low class priors, while maintaining the class priors to be same. Thus the augmentation process is needed.

5.2 Methods

We compared the following methods:

Positive-Unlabeled Set Kernel Classifier (PU-SKC, the proposed method): Hyperparameters (the degree parameter $\rho$ in the polynomial kernel (5) and the regularization parameter $\lambda$ in the objective function (27)) were selected via 5-fold cross-validation from $\rho\in[1,2,3]$ and $\lambda\in[10^{0},10^{-3},10^{-6}]$. Values minimizing the PU risk (16) with the zero-one loss were chosen to be optimal.

• 2.

Least-Squares Density Difference (LSDD): Estimate the density difference $p(\bm{x})-p^{\prime}(\bm{x})$ to obtain (32) using the least-squares method [34]. The bag classifier can be obtained as $g(X)=\max_{\bm{x}\in X}d(\bm{x})$. Hyperparameters (the width of the Gaussian kernel $s$ and the regularization parameter $\lambda$) were selected via 5-fold cross-validation from $s\in[10^{-2},10^{-4},10^{-6}]$ and $\lambda\in[10^{0},10^{-3},10^{-6}]$.

• 3.

Direct Sign of Density Difference (DSDD): Estimate the sign of the density difference (32) directly [33]. The bag classifier can be obtained in the same way as LSDD. Hyperparameters (the width of the Gaussian kernel $s$ and the regularization parameter $\lambda$) were selected via 5-fold cross-validation from $s\in[10^{-2},10^{-4},10^{-6}]$ and $\lambda\in[10^{0},10^{-3},10^{-6}]$.

• 4.

puMIL: An SVM-based approach [27]. The detail is described in B.

5.3 Results

Here we show the experimental results and give answers to the research questions.

5.3.1 Classification Performances

Table 3 shows averages with standard deviations of the classification accuracy over $20$ trials under each class prior. Bold faces represent the best methods under each class prior. This was tested by the one-sided t-test with 5% significance level (first the best method was chosen, then other methods were checked whether they are comparable or not by the one-sided t-test). As it can be seen from Table 3, PU-SKC outperforms the existing method puMIL [27] under various class priors. Note that the true class prior in Table 3 means the predefined value for dataset generation (see Algorithm 1), not an estimated class prior during the learning process (see Sect. 3.3).

Overall, the performance of the proposed method decreases as the true class prior becomes higher. This can be confirmed from Eq. (34): since $2\pi^{*}/\sqrt{N_{\mathrm{P}}}$ dominates the upper bound given sufficiently large $N_{\mathrm{U}}$, more positive bags are needed to achieve accurate classification performance compared with a small class prior case. In practice, we could address this issue by collecting more positive bags. Figure 2 shows classification performances under different number of positive bags. Classification performances are improved as the number of positive bags increases.

**A1: The proposed method tends to outperform the baseline and existing methods under various class priors.

**

5.3.2 Computation Time

Next, we compared the execution time between the proposed method and the baseline and existing methods. The result is shown in Figure 3. This result shows that PU-SKC is much more computationally efficient than the baseline and existing methods. Note that the execution time in other class prior values ($\pi=0.2,0.3,\dots,0.7$) is almost the same as the one shown in Figure 3 because both the class prior estimation algorithm shown in Sec 3.3 and the PU-SKC optimization problem (28) are non-iterative methods and their computation complexities do not depend on the value of $\pi$ or its estimated value.

**A2: The proposed method is much more computationally efficient than the baseline and existing methods.

**

6 Conclusion

In this work, we considered a multiple instance learning problem when only positive bags and unlabeled bags are available, which does not require all training bags to be labeled. We proposed a convex method, PU-SKC, to solve PU multiple instance classification. This method is based on the convex formulation of PU classification [23] and the set kernel [20]. PU-SKC performed better than the existing PU multiple instance classification method [27] on benchmark MIL datasets. Furthermore, we confirmed that the proposed method was much more computationally efficient than the baseline and existing methods through the experiment.

Acknowledgement

TS was supported by KAKENHI 15J09111.

Reference

Appendix A Instance-Level Class Prior Estimation

In the research by du Plessis and Sugiyama [36], a class prior estimation method from positive and unlabeled data by partial matching was proposed. This method estimates the class prior $\pi$ by minimizing the Pearson (PE) divergence from positive data distribution $\pi p(\bm{x}|y=1)$ to unlabeled data distribution $p(\bm{x})$:

[TABLE]

It is still possible to estimate $p(\bm{x}|y=1)$ from positive samples and $p(\bm{x})$ from unlabeled samples using, e.g., the kernel density estimation, but du Plessis and Sugiyama [36] empirically showed that such a naive approach often does not produce a good estimator of $\pi$. A better approach is to lower bound the PE divergence and directly maximize it:

[TABLE]

where $r(\bm{x})$ is an arbitrary function. In practice, a linear-in-parameter model based on Gaussian kernel basis functions is used to estimate $r$:

[TABLE]

Our goal is to find the tightest lower bound by maximizing the right-hand side of (35) at first with respect to $r$ and then minimize it with respect to $\pi$. The former maximization problem with the $l_{2}$ regularizer can be written as

[TABLE]

and $\lambda\geq 0$ denotes the regularization parameter. In practice, $H$ and $\bm{h}$ are estimated by the sample averages:

[TABLE]

Using these estimators, Eq. (36) can be reformulated as follows:

[TABLE]

An analytical solution to the above problem can be obtained as follows:

[TABLE]

where $I$ denotes the identity matrix. This leads to the following PE divergence estimator:

[TABLE]

Then the analytical minimizer of $\widehat{\textsf{PE}}$ can be obtained as

[TABLE]

Appendix B Multiple Instance Learning from Positive and Unlabeled Bags

Wu et al. [27] proposed an instance-level method for the PU-MIL problem (called puMIL). As discussed in the previous work [15, 22], the key instance (the most positive instance) in each bag is important in MIL. This is also the case in PU-MIL, but the problem is that we cannot tell the labels of unlabeled bags. To address this issue, Wu et al. [27] introduced the bag confidence, which describes how much confident the given unlabeled bag is negative. Let $\nu_{b}\in[0,1]$ be the confidence of the $b$-th bag. Then the learning problem of an SVM classifier $g(X)=\bm{w}^{\top}\widetilde{\bm{x}}\quad(\widetilde{\bm{x}}\in X)$ under the given $\bm{\nu}$ is formulated as

[TABLE]

where $\widetilde{Y}_{b}$ is the estimated bag label ($\widetilde{Y}_{b}=+1$ for a positive bag and $\widetilde{Y}_{b}=-1$ for a possibly negative bag coming from the set of unlabeled bags) and $\widetilde{\bm{x}}_{b}$ is the key instance in the bag $X_{b}$. $\nu_{b}$ weighs the contribution of $\widetilde{\bm{x}}_{b}$ according to its confidence. For positive bags, the confidence is simply set as $\nu_{b}=1$. The overall training method is summarized as follows.

Initialize $L$ distinct bag confidences $\bm{\nu}^{(1)},\ldots,\bm{\nu}^{(L)}$, where $L$ is the number of the bag confidences. 2. 2.

Extract $N_{\mathrm{P}}$ possibly negative bags from unlabeled bags. 3. 3.

Make a positive margin pool (PMP), which consists of the key instances from positive bags and possibly negative bags. 4. 4.

Solve (37) with using the PMP to obtain an SVM classifier, which is evaluated by the F-measure.

In Step 2, unlabeled bags with the highest confidences are extracted and regarded as possibly negative. Bag confidences are initialized randomly and updated via a genetic algorithm (see Step 4 and (39)).

In Step 3, first the generative distribution of negative instances is estimated by a weighted kernel density estimator [37]:

[TABLE]

where $\widetilde{N}_{\mathrm{N}}$ is the number of all instances included in the extracted bags in Step 2 and $k$ denotes a kernel function and $\widetilde{X}_{b}^{\mathrm{N}}$ is the $b$-th possibly negative bag. Using the estimator (38), we can obtain a PMP by extracting the key instances from bags:

[TABLE]

After building the PMP, an SVM classifier can be obtained in Step 4. We evaluate the given bag confidences $\bm{\nu}$ by calculating the F-measures777 True bag labels are needed to calculate the F-measure, but those of unlabeled bags are unavailable. This point was not discussed in Wu et al. [27], so we assume that possibly negative bags have negative labels and the other bags in the unlabeled set have positive labels when we calculate the F-measure in our experiments. . Here we have $L$ bag confidences, and we update them by a genetic algorithm. First, the bag confidence with the highest F-measure (denoted by $\widehat{\bm{\nu}}$) is cloned to replace the other confidences under the fixed rate $c\in[0,1]$. This process, called the mutation, is caused by

[TABLE]

where $r$ is drawn from the standard normal distribution, $\bm{\nu}_{l}$ is the $l$-th bag confidence, and $f_{l}$ is the F-measure obtained from $\bm{\nu}_{l}$. $\bm{\nu}_{l}$ is to be replaced for $\bm{v}_{l}$ if $\bm{v}_{l}$ produces a better result. We iterate Steps 2 – 4 until the best bag confidence $\widehat{\bm{\nu}}$ converges or the number of iterations reaches the predefined limit.

After the above training steps, the best bag confidence $\widehat{\bm{\nu}}$ is to be obtained. We use $\widehat{\bm{\nu}}$ for the initial bag confidence and iterate Steps 2 – 4 again to obtain a classifier for test prediction888 In test prediction, test bag confidences are not defined. This point was also not discussed in Wu et al. [27], so we set all test bag confidences to 1 in our experiments. . However, the process to extract possibly negative bags from unlabeled bags based on the bag confidences (Step 2) is not shown to converge to the optimal solution in the previous work, while this method experimentally works well.

Appendix C Proof of Theorem 1

.

By McDiarmid’s inequality [38], we obtain

[TABLE]

because $|-g(\cdot)|\leq C_{\bm{\alpha}}C_{\bm{\phi}}$ by the Cauchy-Schwartz inequality (see the definition (33)). This is equivalent to the following inequality with probability at least $1-\frac{\delta}{2}$:

[TABLE]

Similarly, with probability at least $1-\frac{\delta}{2}$,

[TABLE]

noticing that $\sup_{X,X^{\prime}\in\mathscr{X}}|\ell(-g(X))-\ell(-g(X^{\prime}))|\leq C_{\bm{\alpha}}C_{\bm{\phi}}-0=C_{\bm{\alpha}}C_{\bm{\phi}}$ ($\because$ the Lipschitz constant of $\ell$ is $1$) when applying McDiarmid’s inequality.

Now let us move on to the generalization error bound.

[TABLE]

with probability at least $1-\delta$, which concludes Eq. (34) with $C_{\mathcal{G},\delta}=\sqrt{C_{\bm{\alpha}}^{2}C_{\bm{\phi}}^{2}\log\frac{2}{\delta}/2}$. ∎