Label Propagation for Deep Semi-supervised Learning

TL;DR

This paper introduces a transductive label propagation method based on the manifold assumption to enhance semi-supervised deep learning, especially effective with limited labeled data, by iteratively generating pseudo-labels and training neural networks.

Contribution

It adapts classic transductive label propagation to deep learning, leveraging neural embeddings for improved semi-supervised learning in an inductive framework.

Findings

Improved performance on multiple datasets with few labels

Complementary to existing state-of-the-art methods

Effective in low-label regimes

Abstract

Semi-supervised learning is becoming increasingly important because it can combine data carefully labeled by humans with abundant unlabeled data to train deep neural networks. Classic methods on semi-supervised learning that have focused on transductive learning have not been fully exploited in the inductive framework followed by modern deep learning. The same holds for the manifold assumption---that similar examples should get the same prediction. In this work, we employ a transductive label propagation method that is based on the manifold assumption to make predictions on the entire dataset and use these predictions to generate pseudo-labels for the unlabeled data and train a deep neural network. At the core of the transductive method lies a nearest neighbor graph of the dataset that we create based on the embeddings of the same network.Therefore our learning process iterates between these two steps. We improve performance on several datasets especially in the few labels regime and show that our work is complementary to current state of the art.

Tables

Table 1. Table 1: Impact of weights ω i subscript 𝜔 𝑖 \omega_{i} , class weights ζ j subscript 𝜁 𝑗 \zeta_{j} , and pseudo-labeling by diffusion prediction ( 9 ) or network prediction ( 1 ). Error rate is reported on CIFAR-10 with 500 500 500 labels.

Pseudo-labeling	${\omega }_i$	${\zeta }_j$	CIFAR-10
Diffusion (9)			$36.53\pm 1.42$
		✓	$36.17\pm 1.98$
	✓		$33.32\pm 1.53$
	✓	✓	$32.40\pm 1.80$
GTG [8]	✓	✓	$35.20\pm 2.23$
Network (1)	✓	✓	$35.17\pm 2.46$

Table 2. Table 2: Comparison with the state of the art on CIFAR-10. Error rate is reported. “13-layer” network is used. The top part of the table corresponds to training with pseudo-labels, while the bottom part of the table includes methods that are complementary to ours, as shown by the combination of our method with MT. † † \dagger denotes scores reported in prior work.

Dataset	CIFAR-10
Nb. labeled images	500	1000	2000	4000
Fully supervised	$49.08\pm 0.83$	$40.03\pm 1.11$	$29.58\pm 0.93$	$21.63\pm 0.38$
TDCNN [36]†	-	$32.67\pm 1.93$	$22.99\pm 0.79$	$16.17\pm 0.37$
Network prediction (1) + weights	$35.17\pm 2.46$	$23.79\pm 1.31$	$16.64\pm 0.48$	$13.21\pm 0.61$
Ours: Diffusion prediction (9) + weights	$32.40\pm 1.80$	$22.02\pm 0.88$	$15.66\pm 0.35$	$12.69\pm 0.29$
VAT [26]†	-	-	-	$11.36$
$\Pi$ model [23]†	-	-	-	$12.36\pm 0.31$
Temporal Ensemble [23]†	-	-	-	$12.16\pm 0.24$
MT [38]†	-	$27.36\pm 1.30$	$15.73\pm 0.31$	$12.31\pm 0.28$
MT [38]	$27.45\pm 2.64$	$19.04\pm 0.51$	$14.35\pm 0.31$	$11.41\pm 0.25$
MT + Ours	$24.02\pm 2.44$	$16.93\pm 0.70$	$13.22\pm 0.29$	$10.61\pm 0.28$

Table 3. Table 3: Performance comparison on CIFAR-100 and Mini-ImageNet with 4k and 10k labeled images. Error rate is reported. “13-layer” network is used for CIFAR-100 and Resnet-18 is used for Mini-ImageNet. All methods are reproduced by us.

Dataset	CIFAR-100		Mini-ImageNet-top1		Mini-ImageNet-top5
Nb. labeled images	4000	10000	4000	10000	4000	10000
Fully supervised	$55.43\pm 0.11$	$40.67\pm 0.49$	$74.78\pm 0.33$	$60.25\pm 0.29$	$53.07\pm 0.68$	$38.28\pm 0.38$
Ours	$46.20\pm 0.76$	$38.43\pm 1.88$	$70.29\pm 0.81$	$57.58\pm 1.47$	$47.58\pm 0.94$	$36.14\pm 2.19$
MT [38]	$45.36\pm 0.49$	$36.08\pm 0.51$	$72.51\pm 0.22$	$57.55\pm 1.11$	$49.35\pm 0.22$	$32.51\pm 1.31$
MT + Ours	$43.73\pm 0.20$	$35.92\pm 0.47$	$72.78\pm 0.15$	$57.35\pm 1.66$	$50.52\pm 0.39$	$31.99\pm 0.55$

Equations

$${\hat{y}}_i:=\arg \underset{j}{\max }{f}_{\theta }(x_i{)}_j,$$

$$L_s(X_L,Y_L;\theta ):=\sum _{i=1}^l{\ell }_s\left(f_{\theta }(x_i),y_i\right),$$

$$L_p(X_U,{\hat{Y}}_U;\theta ):=\sum _{i=l+1}^n{\ell }_s\left(f_{\theta }(x_i),{\hat{y}}_i\right),$$

$$L_u(X;\theta ):=\sum _{i=1}^n{\ell }_u(f_{\theta }(x_i),f_{\tilde{\theta }}({\tilde{x}}_i)),$$

$$\displaystyle Y_{ij}\mathrel{:=}\left\{\begin{array}[]{ll}1,&\textrm{if }i\in L\wedge y_{i}=j\\ 0,&\textrm{otherwise}.\end{array}\right.\vspace{-5pt}$$

$$Z:=(I-\alpha \mathcal{W}{)}^{-1}Y,\text{vspace}-5pt$$

$${\hat{y}}_i:=\arg \underset{j}{\max }{z}_{ij},\text{vspace}-5pt$$

$$J(Z):=\frac{\alpha }{2}\sum _{i,j=1}^n{w}_{ij}{\Vert \frac{{\mathbf{z}}_i}{\sqrt{d_{ii}}}-\frac{{\mathbf{z}}_j}{\sqrt{d_{jj}}}\Vert }^2+(1-\alpha ){\Vert Y-Z\Vert }_F^2,$$

$$a_{ij}:=\{\begin{array}{ll}[{\mathbf{v}}_i^{\top }{\mathbf{v}}_j{]}_{+}^{\gamma },& \text{if }i\ne j\wedge {\mathbf{v}}_i\in {\text{NN}}_k({\mathbf{v}}_j)\\ 0,& \text{otherwise}\end{array}$$

$$(I-\alpha \mathcal{W})Z=Y,$$

$${\omega }_i:=1-\frac{H({\hat{\mathbf{z}}}_i)}{\log (c)},$$

$$L_w(X,Y_L,{\hat{Y}}_U;\theta )$$

$$+\sum _{i=l+1}^n{\omega }_i{\zeta }_{{\hat{y}}_i}{\ell }_s\left(f_{\theta }(x_i),{\hat{y}}_i\right),$$

Full text

1 Introduction

Modern approaches to many computer vision problems exploit deep neural networks. These are popular for being very efficient and providing great performance at test time. The downside is a requirement of large amounts of training examples, which are labeled either by humans or automatically on proxy tasks.

Visual data are available in large quantities, however, data reliably annotated by humans are still very scarce. Obtaining large amounts of annotated training data for every single task is not only impractical, potentially costly, but it also turns out to be error prone. The low quality of crowd-sourced annotation is a common motivation to minimize the need of annotation.

In the domain of metric learning, promising results have been recently achieved by unsupervised methods for either learning from scratch or fine-tuning a supervised network for domain adaptation, which devise proxy tasks for learning. These tasks exploit the distribution of data in the original space, for instance pairwise relations of training examples  [42], relations between examples and cluster centroids [1], or considering the manifold structure of data [19]. Alternatively, in self-supervised learning, one can take advantage of additional information like spatial layout in images [5, 12] or temporal relation in videos [40, 28]; or mine for such information in unstructured data by algorithmic supervision using conventional methods [13, 30]. However, most such proxy tasks are inferior when directly compared to laboriously annotated data by humans.

In classification, semi-supervised methods attempt to reduce the number of labeled examples, whereby the fully supervised performance on all data acts as an upper bound. In transductive learning [43, 45], label inference restricted to a given set of unlabeled examples is of interest. In inductive learning, the goal is generalization to new unseen data, while the original training data are discarded. This is achieved e.g. by combining classification loss on labeled data with unsupervised objectives on all data, where the latter act as regularization [41, 38]. Or, an existing classifier can be used to assign pseudo-labels [24, 35], which is another form of algorithmic supervision. Using a powerful classifier trained on carefully annotated data can provide high-quality pseudo-labels, opening the door to learning from real unlabeled, large scale data. In such omni-supervised learning [31], the fully supervised performance on the labeled part is actually the lower bound. This only refreshes the interest in inductive semi-supervised methods.

In this paper, we use efficient transductive label propagation [43] to infer pseudo-labels for unlabeled data, which are used to train the classifier. Label propagation is a graph-based method, and in this work the graph is constructed exploiting the embeddings obtained by the classification network itself. Thus, the proposed method alternates between two steps. First, the network is trained from labeled and pseudo-labeled data. The second step uses the embeddings of the network trained in the previous step to construct a nearest neighbor graph. Label propagation is then used to infer pseudo-labels for unlabeled images, as well as a certainty score per image and per class. Training is performed on all data, using certainty-based weights.

We experimentally show on standard datasets that the proposed method outperforms other semi-supervised approaches. The less labeled data is available, the more pronounced the advantage of the proposed approach is.

2 Related work

The literature is rich in the problem of semi-supervised learning (SSL). The reader is advised to see [3] for an extensive overview. The same holds for SSL in image classification [10, 16, 4, 37]. In this section, we mostly restrict the discussion to approaches that use deep learning for SSL and perform the training on a large image collection with mini-batch optimization.

Prior work on semi-supervised deep learning for image classification is divided into two main categories. The first consists of methods, e.g. [15, 23, 34, 38], that add an unsupervised loss term (often called a regularizer) into the loss function. This term is applied to either all images or only the unlabeled ones. Methods in the second category, e.g. [24, 36], assign pseudo-labels to the unlabeled examples. The pseudo-labeled data are then used in training with a supervised loss, such as cross entropy. Both categories use a standard loss term that is trained with supervision from labeled images. A thorough evaluation of SSL deep image classification can be found in Miyato et al. [27].

Our contribution belongs to the second category, and is conceptually and implementation-wise orthogonal to the first. It is therefore straightforward to combine the proposed method with any method from the first category. We do combine it with [38] as shown in Section 5.

Unsupervised loss in deep SSL. Assuming that every training image, labeled or not, belongs to a single category, a natural requirement on the classifier is to make a confident prediction on the training set. This idea was formalized by Sajjadi et al. [35], where the regularizer is designed to minimize the entropy of the network output. Such a loss term is easily combined with other terms. A similar combination is performed for denoising auto-encoders that are applied on all images in an unsupervised manner [32].

A direction attracting a lot of attention is that of consistency loss, where two related cases, e.g. coming from two similar images, or made by two networks with related parameters, are encouraged to have similar network outputs. Sajjadi et al. [34] is the first, to our knowledge, to use a consistency loss between the outputs of a network on random perturbations of the same image. Laine and Aila [23] rather apply consistency between the output of the current network and the temporal average of outputs during training. The state-of-the-art mean teacher (MT) method [38] replaces output averaging by averaging of network parameters. Consistency loss is commonly measured by squared Euclidean distance. The Jensen-Shannon divergence is used instead by Qiao et al. [29], while complementarity of the two networks is enforced via adversarial examples. A similar idea is proposed by Miyato et al. [26].

Pseudo-labeling in deep SSL. Lee [24] uses the current network to infer pseudo-labels of unlabeled examples, by choosing the most confident class. These pseudo-labels are treated like human-provided labels in the cross entropy loss. Its impact is similar to that of entropy minimization [35]; in both cases the network is forced to have more confident predictions. The same principle is adopted by Shi et al. [36], where the authors further add contrastive loss to the consistency loss. Our method is different from all such prior work in that pseudo-labels are inferred by label propagation rather than network predictions.

Label propagation has been extensively used in a transductive setup (see chapter 11 [3]). Recently, Douze et al. [7] perform label propagation on a large image dataset with CNN descriptors for few shot learning. Unseen images are classified via online label propagation, which requires storing the entire dataset, while the network is trained in advance and descriptors are fixed. Our work is different in that we perform label propagation on the training set off-line while training the network, such that inference is possible without accessing the original training set. Learning by association [17] can been seen as two steps of propagation on a constrained bi-partite graph between labeled and unlabeled examples. Graph transduction game (GTG) [9], a form of label propagation, has been used for pseudo-labels [8] as in our work, but in this case the network is pre-trained, the graph remains fixed and there is no weighting mechanism. We compare to this approach in Section 5.

3 Preliminaries

In this section we formulate the semi-supervised learning problem and then we discuss the classifier, different loss functions that are commonly used in prior work, and finally a transductive learning approach that our method is based on. In our experiments we use a convolutional neural network (CNN) to perform image classification, but this formulation applies to any network architecture in any domain.

Problem formulation. We assume a collection of $n$ examples $X\mathrel{:=}(x_{1},\ldots,x_{l},x_{l+1},\ldots,x_{n})$ with $x_{i}\in\mathcal{X}$. The first $l$ examples $x_{i}$ for $i\in L\mathrel{:=}\{1,\dots,l\}$, denoted by $X_{L}$, are labeled according to $Y_{L}\mathrel{:=}(y_{1},\ldots,y_{l})$ with $y_{i}\in C$, where $C\mathrel{:=}\{1,\dots,c\}$ is a discrete label set for $c$ classes. The remaining $u\mathrel{:=}n-l$ examples $x_{i}$ for $i\in U\mathrel{:=}\{l+1,\dots,n\}$, denoted by $X_{U}$, are unlabeled. The goal in SSL is to use all examples $X$ and labels $Y_{L}$ to train a classifier that maps previously unseen samples to class labels.

Classifier. The network takes an input example from $\mathcal{X}$ and produces a vector of class confidence scores. We denote it by $f_{\theta}:\mathcal{X}\rightarrow^{c}$, where $\theta$ are the network parameters. It is conceptually divided in two parts. The first is a feature extraction network $\phi_{\theta}:\mathcal{X}\rightarrow^{d}$ mapping the input to a feature vector, or descriptor. We denote the descriptor of the $i$-th example by $\mathbf{v}_{i}\mathrel{:=}\phi_{\theta}(x_{i})$. The second typically consists of a fully connected (FC) layer applied on top of $\phi_{\theta}$ and followed by softmax, producing a vector of confidence scores. Function $f_{\theta}$ is the mapping from input space directly to confidence scores. The output of the network for the $i$-th example is $f_{\theta}(x_{i})$ and the prediction is the one of maximum confidence score

[TABLE]

where subscript $j$ denotes the $j$-th dimension of the vector.

Supervised loss. In supervised learning, the network is trained by minimizing a supervised loss term of the form

[TABLE]

which applies only to labeled examples in $X_{L}$. Such term is part of the total loss when training a network in a semi-supervised setup [36, 38, 29]. A standard choice for the loss function $\ell_{s}$ in classification is cross-entropy, given by $\ell_{s}(\mathbf{s},y)\mathrel{:=}-\log\mathbf{s}_{y}$ for $\mathbf{s}\in^{c}$ and $y\in C$.

Pseudo-labeling is the process of assigning a pseudo-label $\hat{y}_{i}$ to each example $x_{i}$ for $i\in U$. Denoting by $\hat{Y}_{U}\mathrel{:=}(\hat{y}_{l+1},\dots,\hat{y}_{n})$ the collection of pseudo-labels for $X_{U}$, the following additional pseudo-label loss term applies

[TABLE]

where again $\ell_{s}$ is any supervised loss function like cross-entropy. An example is the approach proposed by Lee [24], who first train network $f_{\theta}$ with (2) and then assign pseudo-labels according to (1) for $i\in U$.

Unsupervised loss is another common alternative where the loss function applies to both labeled and unlabeled examples and encourages consistency under different transformations of the data or the network. The so-called consistency loss [36, 38, 36] is defined as

[TABLE]

where $\tilde{x}_{i}$ refers to a different transformation of example $x_{i}$. Note that according to the standard practice of data augmentation, every forward pass of $x_{i}$ during training is performed under some random transformation. Parameter set $\tilde{\theta}$ is either equal to $\theta$ or any other transformation of it, such as a moving average over the sequence of network updates [38]. A simple choice of $\ell_{u}$ is the squared Euclidean distance, i.e. $\ell_{u}(\mathbf{s},\tilde{\mathbf{s}})\mathrel{:=}||\mathbf{s}-\tilde{\mathbf{s}})||^{2}$ for $\mathbf{s},\tilde{\mathbf{s}}\in^{c}$, forcing the two outputs to be as close as possible.

Transductive learning solves a more specific problem. Instead of training a generic classifier able to classify new, yet unseen, examples, the goal is to use $X$ and $Y_{L}$ to infer labels for examples in $X_{U}$. In this work, we adopt the graph-based approach of Zhou et al. [43] for transductive learning by diffusion111We first present the original approach and discuss our design choices in the following section..

Diffusion for transductive learning [43]. Let $V=(\mathbf{v}_{1},\ldots,\mathbf{v}_{l},\mathbf{v}_{l+1},\ldots,\mathbf{v}_{n})$ be the descriptor set, where $\mathbf{v}_{i}$ corresponds to $x_{i}$ as defined earlier. A symmetric adjacency matrix $W\in^{n\times n}$ with zero diagonal is constructed, whose elements $w_{ij}$ are non-negative pairwise similarities between $\mathbf{v}_{i}$ and $\mathbf{v}_{j}$. Its symmetrically normalized counterpart is given by $\mathcal{W}=D^{-1/2}WD^{-1/2}$, where $D\mathrel{:=}\operatorname{diag}(W\mathbf{1}_{n})$ is the degree matrix and $\mathbf{1}_{n}$ is the all-ones $n$-vector. A $n\times c$ label matrix $Y$ is defined with elements

[TABLE]

That is, the rows of $Y$ corresponding to labeled examples are one-hot encoded labels and the rest are zero. Diffusion amounts to computing the $n\times c$ matrix

[TABLE]

where $\alpha\in[0,1)$ is a parameter. Finally, the class prediction for an unlabeled example $x_{i}$ is

[TABLE]

where $z_{ij}$ is the $(i,j)$ element of matrix $Z$.

It is interesting to observe that matrix $Z$ as defined by (8) is the minimizer of the following quadratic cost function

[TABLE]

where $\mathbf{z}_{i}$ is the $i$-th row of matrix $Z$, $d_{ii}$ is the $i$-th diagonal diagonal element of $D$ and $\left\|{\cdot}\right\|_{F}$ is the Frobenius norm. The first term encourages smoothness such that nearby examples get the same predictions, while the second attempts to maintain predictions for the labeled examples [43].

4 Method