Kernel Mean Embedding of Instance-wise Predictions in Multiple Instance Regression

TL;DR

This paper introduces an improved multiple instance regression method that uses kernel mean embeddings of predicted label distributions, leading to better performance than previous approaches on real-world datasets.

Contribution

The paper extends the instance-MIR algorithm by incorporating kernel mean embeddings to better capture label distribution information within bags.

Findings

Outperforms baseline instance-MIR on five datasets

Achieves state-of-the-art results on two datasets

Provides a more expressive representation of bag label distributions

Abstract

In this paper, we propose an extension to an existing algorithm (instance-MIR) which tackles the multiple instance regression (MIR) problem, also known as distribution regression. The MIR setting arises when the data is a collection of bags, where each bag consists of several instances which correspond to the same and unique real-valued label. The goal of a MIR algorithm is to find a mapping from the instances of an unseen bag to its target value. The instance-MIR algorithm treats all the instances separately and maps each instance to a label. The final bag label is then taken as the mean or the median of the predictions for that given bag. While it is conceptually simple, taking a single statistic to summarize the distribution of the labels in each bag is a limitation. In spite of this performance bottleneck, the instance-MIR algorithm has been shown to be competitive when compared to the current state-of-the-art methods. We address the aforementioned issue by computing the kernel mean embeddings of the distributions of the predicted labels, for each bag, and learn a regressor from these embeddings to the bag label. We test our algorithm (instance-kme-MIR) on five real world datasets and obtain better results than the baseline instance-MIR across all the datasets, while achieving state-of-the-art results on two of the datasets.

Tables

Table 1. Table 1. The loss for the 3 AOD datasets (MISR1, MISR2, MODIS) is the RMSE × \times 100 and for the 2 CROP datasets (WHEAT, CORN) the loss is the RMSE.

	Datasets
Algorithms	MODIS	MISR1	MISR2	WHEAT	CORN
Instance-MIR (mean)	10.4	9.02	7.61	4.96	24.57
Instance-MIR (median)	10.4	8.89	7.50	5.00	24.72
Instance-kme-MIR ($k_{\text{INV}}$)	10.1	8.68	7.28	4.91	24.40
Instance-kme-MIR ($k_{\text{RBF}}$)	10.1	8.70	7.38	4.90	24.51
EM-MIR (Wang et al., 2012)	9.5	7.5	7.3	4.9	26.8
Attention-MIR (Uriot, 2019)	9.05	7.32	6.95	5.24	27.00

Equations

$$\hat{f}={\text{arg min}}_{f\in \mathcal{H}}\frac{1}{B^{\ast }}\sum _{i=1}^{B^{\ast }}MSE(y_i^{\ast },f(\{x_{i,l}^{\ast }{\}}_{l=1}^{L_i^{\ast }})),$$

$${\hat{y}}_i^{\ast }=\frac{1}{L_i^{\ast }}\sum _{l=1}^{L_i^{\ast }}{\hat{y}}_{i,l}^{\ast },i=1\dots B^{\ast },$$

$${\mu }_{\text{P}}=\int k(\cdot ,x)\text{P}(dx)\in {\mathcal{H}}_k.$$

$${\hat{\mu }}_{{\text{P}}_i}=\int k(\cdot ,x){\hat{\text{P}}}^i(dx)=\frac{1}{L_i}\sum _{l=1}^{L_i}k(\cdot ,x_{i,l}),\text{for bag i}.$$

$$\hat{\alpha }=\underset{\alpha }{\mathrm{argmin}}(\Vert y-\text{K}\alpha {\Vert }^2+\lambda {\alpha }^T\text{K}\alpha ),$$

$$\text{K}(i,j)=k^{{}^{\prime }}({\hat{\mu }}_{{\text{P}}_i},{\hat{\mu }}_{{\text{P}}_j}),\text{for bags i,j=1,…,B,}$$

$$\text{K}(i,j)=$$

$$\displaystyle=\Big{<}\frac{1}{L_{i}}\sum_{l=1}^{L_{i}}k(\cdot,x_{i,l}),\frac{1}{L_{j}}\sum_{m=1}^{L_{j}}k(\cdot,x_{j,m})\Big{>}_{\mathcal{H}_{k}}$$

$$=\frac{1}{L_i}\frac{1}{L_j}\sum _{l=1}^{L_i}\sum _{m=1}^{L_j}k(x_{i,l},x_{j,m}),$$

$${\hat{y}}_{test}=\hat{\alpha }{\text{K}}_{test},$$

$$k(x,x^{\prime }{)}_{RBF}=\text{exp}\left(-\frac{\Vert x-x^{\prime }{\Vert }^2}{2{\theta }^2}\right),k(x,x^{\prime }{)}_{INV}=\frac{1-\Vert x-x^{\prime }{\Vert }^2}{\Vert x-x^{\prime }{\Vert }^2+\theta }.$$

Full text

Distribution regression; Kernel mean embedding; Instance-wise regression; Multiple instance regression

††journalyear: 2019††copyright: acmlicensed††conference: The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; August 03–07, 2019; Anchorage, Alaska USA††booktitle: KDD ’19: The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, August 03–07, 2019, Anchorage, Alaska USA

1. Introduction

Multiple instance learning (MIL) is a setting which falls under the supervised learning paradigm. Within the MIL framework, there exist two different learning tasks: multiple instance classification (MIC) (Amores, 2013) and multiple instance regression (MIR) (Dietterich et al., 1997; Ray and Page, 2001). The former has been extensively studied in the literature while the latter has been underrepresented. This could be due to the fact that many of the data sources studied within the MIL framework are images and text, which correspond to classification tasks. The MIC problem generally consists in classifying bags into positive or negative examples where negative bags contain only negative instances and positive bags contain at least one positive instance. A multitude of applications are covered by the MIC framework. It has been applied to medical imaging in a weakly supervised setting (Wu et al., 2015; Xu et al., [n. d.]) where each image is taken as a bag and sub-regions of the image are instances, to image categorization (Chen and Wang, 2004) and retrieval (Zhang et al., 2002; Yang and Lozano-Perez, 2000) and to analyzing videos (Sikka et al., 2013), where the video is treated as the bag and the frames are the instances.

On the other hand, the MIR problem, where bags labels are now real valued, has been much less prevalent in the literature. In a regression setting, as opposed to classification, one cannot simply identify a single positive instance. Instead, one needs to estimate the contribution of each of the instances towards the bag label. The MIR problem was first introduced in the context of predicting drug activity level (Dietterich et al., 1997) and the first proposed MIR algorithm relied on the assumption that the bag’s label can be fully explained by a single prime instance (prime-MIR) (Ray and Page, 2001). However, this is a simplistic assumption as we throw away a lot of information about the distribution (e.g, variance, skewness). Instead of assuming that a single instance is responsible for the bag’s label, the MIR problem has been tackled using a weighted linear combination of the instances (Wagstaff and Lane, 2007), or as a prime cluster of instances (cluster-MIR) (Wagstaff et al., 2008). Other works have looked at first efficiently mapping the instances in each bag to a new embedding space, and then train a regressor on the new embedded feature space. For instance, one can transform the MIR problem to a regular supervised learning problem by mapping each bag to a feature space which is characterized by a similarity measure between a bag and an instance (Chen et al., 2006). The resulting embedding of a bag in the new feature space represents how similar a bag is to various instances from the training set. A drawback of this approach is that the embedding space for each bag can be high-dimensional when the number of instances in the training set is large, producing many redundant and possibly uninformative features.

In this paper, we use a similar approach and compute the kernel mean embeddings for each bag (Muandet et al., 2017). The use of kernel mean embedding in distribution regression has been applied to various real-world problems such as analyzing the 2016 US presidential election (Flaxman et al., 2016b) and estimating aerosol levels in the atmosphere (Szabó et al., 2015). Intuitively, kernel mean embedding measures how similar each bag is to all the other bags from the training set. In this paper, as opposed to previous works, we do not compute the kernel mean embeddings directly on the input features (i.e, on the instances) but on the predictions made by a previous learning algorithm (e.g, a neural network). This insight comes from the fact that a simple baseline algorithm (instance-MIR) performed surprisingly well on several datasets, when the regressor was a neural network with a large hidden layer (Uriot, 2019). The instance-MIR algorithm essentially ignores the fact that we are in a distribution regression framework and treats each instance as a separate observation, thereby yielding a unique prediction for each instance. The final bag label is taken to be the mean or the median of the predictions for that given bag. However, using a point estimate in the original prediction space is a performance bottleneck. In this paper, we propose a novel algorithm (instance-kme-MIR), which leverages both the representational power of the instance-MIR algorithm equipped with a neural network and alleviate the aforementioned issue by mapping our predictions into a high or infinite-dimensional space, characterized by a kernel function. We test our approach on 5 remotely sensed real-world datasets.

2. Related Work

The datasets we are using to test our algorithm stems from remotely sensed data111http://www.dabi.temple.edu/~vucetic/MIR.html 222https://harvist.jpl.nasa.gov/papers.shtml, and have previously been described (Uriot, 2019; Wang et al., 2012) and studied as a distribution regression problem (Wang et al., 2008; Wang et al., 2012; Uriot, 2019). This allows us to compare the performance of our approach with the baseline instance-MIR and the current state-of-the-art. The first application (3 of the 5 datasets) consists in predicting aerosol optical depth (AOD) - aerosols are fine airborne solid particles or liquid droplets in air, that both reflect and absorb incoming solar radiation. The second application (2 of the 5 datasets) is the prediction of county-level crop yields (Wagstaff and Lane, 2007) (wheat and corn) in Kansas between 2001 and 2005. These two applications can naturally be framed as a multiple instance regression problem. Indeed, in both applications, satellites will gather noisy measurements due to the intrinsic variability within the sensors and the properties of the targeted area on Earth (e.g, surface and atmospheric effects). For the AOD prediction task, aerosols have been found to have a very small spatial variability over distances up to 100 km (Ichoku et al., 2002). For the crop data, we can reasonably assume that the yields are similar across a county and thus consider the bag label as the aggregated yield over the entire county.

The first study which investigated estimating AOD levels within a MIR setting, proposed an iterative method (pruning-MIR) which prunes outlying instances from each bag and then proceeds in a similar fashion as instance-MIR (Wang et al., 2008). The main drawback of this approach is that it is not obvious what the pruning threshold should be and we may thus get rid of informative instances in the process. In a subsequent work, the authors investigated a probabilistic framework (EM-MIR) by fitting a mixture model and using the expectation-maximization (EM) algorithm to learn the mixing and distribution parameters (Wang et al., 2012). The current state-of-the-art algorithm (attention-MIR) on the AOD datasets has been obtained by treating each bag as a set (i.e, an unordered sequence) of instances (Uriot, 2019). To do so, the authors implemented an order-invariant operation characterized by a content-based attention mechanism (Vinyals et al., 2015), which then attends the instances a selected number of times. Finally, the problem of estimating AOD levels has been tackled using kernel mean embedding directly on the input features (i.e, the instances) (Szabó et al., 2015), where they show that performance is robust to the kernel choice but the hyperparameter values of the kernels are of primary importance. In this paper, however, we compute the kernel mean embeddings of the distributions of the predicted labels made by a neural network. In order to have a principled way to find the kernel parameters, authors have proposed a Bayesian kernel mean embedding model with a Gaussian process prior, from which we can obtain a closed form marginal pseudolikelihood (Flaxman et al., 2016a). This marginal likelihood can then be optimized in order to find the kernel parameters.

3. Background

3.1. Multiple Instance Regression

In the MIR problem, our observed dataset is $\{(\{x_{i,l}\}_{l=1}^{L_{i}},y_{i})\}_{i=1}^{B}$, where B is the number of bags, $y_{i}\in\mathbb{R}$ is the label of bag $i$, $x_{i,l}$ is the $l^{th}$ instance of bag $i$ and $L_{i}$ is the number of instances in bag $i$. Note that $x_{i,l}\in\mathcal{X}$, and $\mathcal{X}$ is a subset of $\mathbb{R}^{d}$, where $d$ is the number of features in each instance. The number of features must be the same for all the instances, but the number of instances can vary within each bag.

We want to learn the best mapping $\hat{f}$: $\{x_{i,l}\}_{l=1}^{L_{i}}\to\hat{y}_{i}$, $i=1\ldots B$. By best mapping we mean the function $\hat{f}$ which minimizes the mean squared error (MSE) on bags unseen during training (e.g, on the validation set). Formally, we seek $\hat{f}$ such that

[TABLE]

from the validation data $\{(\{x_{i,l}^{*}\}_{l=1}^{L_{i}^{*}},y_{i}^{*})\}_{i=1}^{B^{*}}$, where $\mathcal{H}$ is the hypothesis space of functions $f$ under consideration.

The two main challenges that the multiple instance regression problem poses are to find which instances are predictive of the bag’s label and to efficiently summarize the information from the instances within each bag. However, the instance-MIR baseline algorithm, which we describe next, does not attempt to solve the multiple instance regression problem by addressing the two aforementioned challenges. Instead, it simply treats each instance independently and fit a regression model to all the instances separately.

3.2. Instance-MIR Algorithm

As mentioned, the instance-MIR algorithm makes predictions on all the instances before taking the mean or the median of the predictions for each bag, as the final prediction. This means that during training, all the instances have the same weights and thus contribute equally to the loss function.

Formally, our dataset is formed by pairs of instance and bag label which can be denoted as $\{(x_{i,l},y_{i}),\hskip 2.84544pti=1\ldots B,\hskip 2.84544ptl=1\ldots L_{i}\}$. The final label prediction on an unseen bag can be simply calculated as

[TABLE]

where $\hat{y}_{i,l}^{*}$ is the predicted label corresponding to the $l_{th}$ instance in bag $i$. Empirically, this method has been shown to be competitive (Ray and Craven, 2005), even though it requires models with high complexity in order to be able to effectively map many different noisy instances to the same target value. Thus, it is appropriate to take $\hat{f}$ as a neural network with a large number of hidden units (Uriot, 2019), as apposed to a small number (Wang et al., 2012).

3.3. Kernel Mean Embedding

In this subsection, we briefly describe kernel mean embedding and its application to distribution regression, where the goal is to compute the kernel mean embedding of each bag. We assume that the instances $\{x_{i,l}\}_{l=1}^{L_{i}}$ in each bag, are i.i.d. samples from some unobserved distribution $P^{i}$, for $i=1,\ldots,B$. The idea is to adopt a two-stage procedure by first representing each set of samples (i.e, bags) $\{x_{i,l}\}_{l=1}^{L_{i}}$ by its corresponding kernel mean embedding and then train a kernel ridge regression on those embeddings (Szabó et al., 2015).

Formally, let $\mathcal{H}_{k}$ be a reproducing kernel Hilbert space (RKHS), which is a potentially infinite dimensional space of functions $f:\mathcal{X}\to\mathbb{R}$, and let $k:\mathcal{X}\times\mathcal{X}\to\mathbb{R}$ be a reproducing kernel function of $\mathcal{H}_{k}$. Then for $f\in\mathcal{H}_{k},x\in\mathcal{X}$, we can evaluate $f$ at $x$ as an inner product $f(x)=\langle f,k(\cdot,x)\rangle_{\mathcal{H}_{k}}$ (reproducing kernel property). Then, for a probability measure $\textrm{P}\in\mathcal{X}$ we can define its kernel mean embedding as

[TABLE]

For $\mu_{P}$ to be well-defined, we simply require that the norm of $k$ is finite, and so we want $k(\cdot,x)$ such that $\int\sqrt{k(x,x)}\textrm{P}(dx)<\infty$. This is always true for kernel functions that are bounded (e.g, Gaussian RBF, inverse multiquadric) but may be violated for unbounded ones (e.g, polynomial) (Szabó et al., 2015). In fact, it has been shown that the kernel mean embedding approach to distribution regression does not yield satisfying results when using a polynomial kernel, due to the aforementioned violation (Szabó et al., 2015).

However, as mentioned, we do not have access to $\textrm{P}^{i}$ but only observe i.i.d. samples $\{x_{i,l}\}_{l=1}^{L_{i}}$ drawn from it. Instead, we compute the empirical mean estimator $\hat{\mu}_{P}$ of $\mu_{P}$, given by

[TABLE]

3.4. Kernel Ridge Regression

In kernel ridge regression (KRR), we seek to find the set of parameters $\hat{\alpha}$, such that

[TABLE]

where $\textbf{K}\in\mathbb{R}^{n\times n}$ is the kernelized Gram matrix of the dataset, and $\lambda$ is the hyperparameter controlling the amount of weight decay (i.e, $L_{2}$ regularization) on the parameters $\alpha$. In the case of KRR applied to kernel mean embedding, we have

[TABLE]

where $k^{{}^{\prime}}$ is the KRR kernel and $\textbf{K}\in\mathbb{R}^{B\times B}$ ($B$ is the number of bags in the training set). In this paper, we take $k^{{}^{\prime}}$ to be the linear kernel, as it simplifies the computation and has been shown to yield competitive results when compared to non-linear kernels (Szabó et al., 2015). Thus, we have that

[TABLE]

where $x_{i,l}$ is the $l^{th}$ instance of bag $i$ and $x_{j,m}$ is the $m^{th}$ instance of bag j and the last equality is due to the reproducing property. In order to make predictions $\hat{y}_{test}$ on bags not seen during training, we simply compute

[TABLE]

where $\hat{\alpha}=y_{train}(\textbf{K}_{train}+\lambda\textbf{I}_{B\times B})^{-1}$ is obtained by differentiating (4) with respect to $\alpha$, equating to 0, and solving for $\alpha$. Note that as mentioned in subsection 3.1, $B$ is the number of bags in the training set and $B^{*}$ is the number of unseen bags (e.g, in validation or testing set), and so $\textbf{K}_{test}\in\mathbb{R}^{B\times B^{*}}$.

4. Instance-kme-MIR Algorithm

In this section, we describe our novel algorithm (instance-kme-MIR), and discuss the choice we made for the hyperparameter values. We emphasise that the novelty in this paper is to compute the kernel mean embeddings on the predictions made by a previous learning algorithm, as opposed to previous works (Szabó et al., 2015; Law et al., 2017; Flaxman et al., 2016b; Flaxman et al., 2016a), where the authors directly compute the kernel mean embeddings on the input features. Our algorithm can be seen as an extension of instance-MIR, where we take advantage of the representational power of neural networks (Part 1 of our algorithm), and address its performance bottleneck by computing the kernel mean embeddings on the predictions (Part 2 of our algorithm).

In our implementation333 https://github.com/pinouche/Instance-kme-MIR, we choose $D=50$ and $f$ to be a single layered neural network, as it was shown to yield good results for the instance-MIR algorithm (Uriot, 2019). We purposefully set the number of folds $D$ to be large, so that in Part 1 of our algorithm, we still train $f$ on $98\%$ of the training set. It thus makes sense to use the same hyperparameter values for the neural network $f$ when comparing the baseline instance-MIR to instance-kme-MIR. For Part 2, we experimented with two different kernels $k$ (RBF and inverse multiquadric).

5. Evaluation

5.1. Training Protocol

In order to fairly compare our algorithm to the current state-of-the-art (Uriot, 2019; Wang et al., 2012), we evaluate its performance using the same training and evaluation protocol. The protocol consists in a 5-fold cross validation, where the bags in the training set are randomly split into 5 folds, out of which 4 folds are used in training and 1 fold serves as the validation set. In turn, each of the 5 folds serves as the validation set and the 4 remaining folds as the training set. The cross validation is repeated 10 times in order to eliminate the randomness involved in choosing the folds. We use the root mean squared error (RMSE) to evaluate the performance and report our results, shown in Table 1, on 5 real-world datasets. While the baseline instance-MIR was already evaluated (Uriot, 2019), we re-implement it on the 3 AOD datasets, with different hyperparameter values, and thus obtain distinct results. The validation loss reported in Table 1 below is the average loss over the 50 evaluations (10 iterations of 5-fold cross validation).

5.2. Results

In Table 1, we display the results for 4 algorithms: the baseline instance-MIR (described in subsection 3.2), attention-MIR (Uriot, 2019), EM-MIR (Wang et al., 2012) and our novel algorithm (instance-kme-MIR), for two different kernels $k_{\textrm{RBF}}$ and $k_{\textrm{INV}}$, where

[TABLE]

Note that prior to our implementation of instance-kme-MIR, the state-of-the-art results on the 5 datasets were shared between the 3 other algorithms (Uriot, 2019). Now, as can be seen in Table 1, attention-MIR achieves the best results on the AOD datasets while instance-kme-MIR yields the best results on the crop datasets.

We experimented with several values for $\theta$ and $\lambda$, where $\theta\in\{10,20,\ldots,130,140\}$ and $\lambda\in\{10^{-1},10^{-2},\ldots,10^{-15},10^{-16}\}$, with a constant increment for both hyperparameters. The results in Table 1 are reported for the best hyperparameter values. We found that while extreme hyperparameter values negatively impacted the performance of our algorithm, most values yielded similarly good results, which means that our algorithm is robust to hyperparameter values.

Instance-MIR (median) refers to the instance-MIR algorithm where the median is used to compute the final prediction for each bag, instead of the mean, as described in subsection 3.2. We can see that there does not seem to be an advantage to using the mean or the median, as both methods achieve very similar results. On the other hand, we can see that our algorithm consistently outperforms the baseline instance-MIR. However, note that since our algorithm makes use of the predictions made from instance-MIR (in Part 2 of Algorithm 1), we can only aim to achieve a measured improvement over the standard instance-MIR. Thus, our method is mostly beneficial in the cases where instance-MIR is the best out-of-the box algorithm (e.g, on the 2 crop datasets). Since our algorithm computes the kernel mean embedding between scalars (i.e, between the real-valued predictions) and is robust to values of $\lambda$ and $\theta$, it is easy to tune and its computation cost is very close to that of instance-MIR.

6. Conclusion

In this paper, we developed a straightforward extension of the baseline instance-MIR algorithm. Our method takes advantage of the expressive power of neural networks while addressing the main weakness of instance-MIR by computing the kernel mean embeddings of the predictions. We have shown that our algorithm consistently outperforms the baseline and achieves state-of-the-art results on the 2 crop datasets. In addition, our algorithm is robust to the kernel parameter values and its performance gains come at a low computational cost.

Nonetheless, it fails when the baseline instance-MIR does not yield satisfying results (e.g, on the 3 crop datasets). This is because we compute the kernel mean embeddings on predictions made from the baseline instance-MIR, and we can thus only expect measured improvements from that baseline. Another drawback of our method comes from the fact that instance-MIR assigns the same weights to all the instances during training. However, the number of instances per bag may vary and it would make sense to be more confident when we make a prediction on a bag which contains a large number of instances compared to a bag with only a few instances. To tackle this issue, we could take a Bayesian approach to kernel mean embedding and explicitly express our uncertainty in the sampling variability of the groups (Law et al., 2017).

Finally, as future work, we could use the attention coefficients from attention-MIR, in order to weigh the contribution of each of the instances towards the loss function. This would get rid of potentially redundant and noisy instances, thus improving the quality of the training data.