An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning

TL;DR

This paper introduces E3BM, a meta-learned ensemble of epoch-wise empirical Bayes models, to improve robustness and efficiency in few-shot learning across multiple benchmarks.

Contribution

It proposes a novel meta-learning approach with four hyperprior learners for epoch-wise ensembling in few-shot tasks, achieving top performance.

Findings

Top performance on miniImageNet, tieredImageNet, and FC100.

Epoch-dependent transductive hyperprior yields best results.

Both epoch-wise ensembling and empirical methods enhance robustness.

Abstract

Few-shot learning aims to train efficient predictive models with a few examples. The lack of training data leads to poor models that perform high-variance or low-confidence predictions. In this paper, we propose to meta-learn the ensemble of epoch-wise empirical Bayes models (E3BM) to achieve robust predictions. "Epoch-wise" means that each training epoch has a Bayes model whose parameters are specifically learned and deployed. "Empirical" means that the hyperparameters, e.g., used for learning and ensembling the epoch-wise models, are generated by hyperprior learners conditional on task-specific data. We introduce four kinds of hyperprior learners by considering inductive vs. transductive, and epoch-dependent vs. epoch-independent, in the paradigm of meta-learning. We conduct extensive experiments for five-class few-shot tasks on three challenging benchmarks: miniImageNet, tieredImageNet, and FC100, and achieve top performance using the epoch-dependent transductive hyperprior learner, which captures the richest information. Our ablation study shows that both "epoch-wise ensemble" and "empirical" encourage high efficiency and robustness in the model performance.

Tables

Table 1. Table 1: The 5 5 5 -class few-shot classification accuracies (%) on mini ImageNet, tiered ImageNet, and FC100. “( + time, + param)” denote the additional computational time (%) and parameter size (%), respectively, when plugging-in E 3 BM to baselines (MAML, MTL and SIB). “–” means no reported results in original papers. The best and second best results are highlighted.

Methods	Backbone	miniImageNet			tieredImageNet			FC100
		$1$-shot	$5$-shot		$1$-shot	$5$-shot		$1$-shot	$5$-shot
MatchNets [73]	4CONV	43.44	55.31		–	–		–	–
ProtoNets [64]	4CONV	49.42	68.20		53.31	72.69		–	–
MAML⋄ [13]	4CONV	48.70	63.11		49.0	66.5		38.1	50.4
MAML++⋄ [1]	4CONV	52.15	68.32		51.5	70.6		38.7	52.9
TADAM [53]	ResNet-12	58.5	76.7		–	–		40.1	56.1
MetaOptNet [37]	ResNet-12	62.64	78.63		65.99	81.56		41.1	55.5
CAN [24]	ResNet-12	63.85	79.44		69.89	84.23		–	–
CTM [40]	ResNet-18	64.12	80.51		68.41	84.28		–	–
MTL [70]	ResNet-12	61.2	75.5		–	–		45.1	57.6
MTL⋄ [70]	ResNet-25	63.4	80.1		69.1	84.2		43.7	60.1
LEO [61]	WRN-28-10	61.76	77.59		66.33	81.44		–	–
Robust20-dist‡ [12]	WRN-28-10	63.28	81.17		–	–		–	–
MAML+E3BM	4CONV	53.2($\uparrow$4.5)	65.1($\uparrow$2.0)		52.1($\uparrow$3.1)	70.2($\uparrow$3.7)		39.9($\uparrow$1.8)	52.6($\uparrow$2.2)
(+time, +param)	–	(8.9, 2.2)	(9.7, 2.2)		(10.6, 2.2)	(9.3, 2.2)		(7.8, 2.2)	(12.1, 2.2)
MTL+E3BM	ResNet-25	64.3($\uparrow$0.9)	81.0($\uparrow$0.9)		70.0($\uparrow$0.9)	85.0($\uparrow$0.8)		45.0($\uparrow$1.3)	60.5($\uparrow$0.4)
(+time, +param)	–	(5.9, 0.7)	(10.2, 0.7)		(6.7, 0.7)	(9.5, 0.7)		(5.7, 0.7)	(7.9, 0.7)
(a) Inductive Methods
EGNN [31]	ResNet-12	64.02	77.20		65.45	82.52		–	–
CAN+T [24]	ResNet-12	67.19	80.64		73.21	84.93		–	–
SIB⋄‡ [25]	WRN-28-10	70.0	79.2		72.9	82.8		45.2	55.9
SIB+E3BM‡	WRN-28-10	71.4($\uparrow$1.4)	81.2($\uparrow$2.0)		75.6($\uparrow$2.7)	84.3($\uparrow$1.5)		46.0($\uparrow$0.8)	57.1($\uparrow$1.2)
(+time, +param)	–	(2.1, 0.04)	(5.7, 0.04)		(5.2, 0.04)	(4.9, 0.04)		(6.1, 0.04)	(7.3, 0.04)
(b) Transductive Methods
⋄Our implementation on tieredImageNet and FC100.   ‡Input image size: $80\times 80\times 3$.

Table 2. Table 2: The 5 5 5 -class few-shot classification accuracies (%) of using different hyperprior learners, on the mini ImageNet, tiered ImageNet, and FC100. “Ind.” and “Tra.” denote the inductive and transductive settings, respectively. The best and second best results are highlighted.

No.	Setting				miniImageNet			tieredImageNet			FC100
	Method	Hyperprior	Learning		$1$-shot	$5$-shot		$1$-shot	$5$-shot		$1$-shot	$5$-shot
1	MTL [70]	–	Ind.		63.4	80.1		69.1	84.2		43.7	60.1
2	MTL+E3BM	FC	Ind.		64.3	80.9		69.8	84.6		44.8	60.5
3	MTL+E3BM	FC	Tra.		64.7	80.7		69.7	84.9		44.7	60.6
4	MTL+E3BM	LSTM	Ind.		64.3	81.0		70.0	85.0		45.0	60.4
5	MTL+E3BM	LSTM	Tra.		64.5	81.1		70.2	85.3		45.1	60.6
6	SIB [25]	–	Tra.		70.0	79.2		72.9	82.8		45.2	55.9
7	SIB+E3BM	FC	Tra.		71.3	81.0		75.2	83.8		45.8	56.3
8	SIB+E3BM	LSTM	Tra.		71.4	81.2		75.6	84.3		46.0	57.1

Table 3. Table S1: Supplementary to Table  1 . Few-shot classification accuracy (%) on mini ImageNet, tiered ImageNet, and FC100 ( 5 5 5 -class).

Methods	Backbone	miniImageNet			tieredImageNet			FC100
		$1$-shot	$5$-shot		$1$-shot	$5$-shot		$1$-shot	$5$-shot
MAML+E3BM	4CONV	53.2 $\pm$ 1.8	65.1 $\pm$ 0.9		52.1 $\pm$ 1.8	70.2 $\pm$ 0.9		39.9 $\pm$ 1.8	52.6 $\pm$ 0.9
MTL+E3BM	ResNet-12	63.8 $\pm$ 0.4	80.1 $\pm$ 0.3		71.2 $\pm$ 0.4	85.3 $\pm$ 0.3		43.2 $\pm$ 0.3	60.2 $\pm$ 0.3
MTL+E3BM	ResNet-25	64.3 $\pm$ 0.4	81.0 $\pm$ 0.3		70.0 $\pm$ 0.4	85.0 $\pm$ 0.3		45.0 $\pm$ 0.4	60.5 $\pm$ 0.3
SIB+E3BM	WRN-28-10	71.4 $\pm$ 0.5	81.2 $\pm$ 0.4		75.6 $\pm$ 0.6	84.3 $\pm$ 0.4		46.0 $\pm$ 0.6	57.1 $\pm$ 0.4

Table 4. Table S2: Supplementary to Table  2 . Results (%) for different hyperprior learners on mini ImageNet, tiered ImageNet, and FC100 ( 5 5 5 -class). “Ind.” and “Tra.” denote inductive and transductive settings, respectively.

No.	Setting				miniImageNet			tieredImageNet			FC100
	Method	Hyperprior	Learning		$1$-shot	$5$-shot		$1$-shot	$5$-shot		$1$-shot	$5$-shot
1	MAML [13]	–	Ind.		48.70	63.11		49.0	66.5		38.1	50.4
2	5x MAML	–	Ind.		52.1	65.1		51.1	68.8		40.1	50.8
3	MAML+E3BM	FC	Ind.		52.1	65.1		51.1	68.8		39.5	51.7
4	MAML+E3BM	FC	Tra.		52.8	65.3		52.2	69.5		40.4	51.8
5	MAML+E3BM	LSTM	Ind.		53.2	65.0		52.1	70.2		39.9	52.6
6	MAML+E3BM	LSTM	Tra.		53.8	65.2		52.7	70.5		40.4	52.3

Table 5. Table S3: Supplementary to Table  1 . The inference time and the number of parameters of baselines ( 100 100 100 epochs, mini ImageNet, 5 5 5 -way 1 1 1 -shot, on NVIDIA V100 GPU).

No.	Method	Backbone	# Param	Time (min)
1	MTL	ResNet-25	4,321k	73.3
2	MTL+E3BM (ours)	ResNet-25	4,351k	77.6
3	SIB	WRN-28-10	36,475k	325.0
4	SIB+E3BM (ours)	WRN-28-10	36,490k	331.8
5	ProtoNets	ResNet-12	12,424k	35.2
6	MatchNets	ResNet-12	12,424k	37.3
7	ProtoNets	ResNet-25	36,579k	70.5
7	ProtoNets	WRN-28-10	36,482k	350.1

Equations

$$p(\mathcal{T})=\prod _{k=1}^{K}p({\mathcal{T}}_k)=\prod _{k=1}^K{\int }_{{\psi }_k}p({\mathcal{T}}_k|{\psi }_k)p({\psi }_k)d{\psi }_k.$$

$$\log p(\mathcal{T})\geqslant\sum_{k=1}^{K}\Big{[}\mathbb{E}_{\psi_{k}\sim q_{\varphi_{k}}}\big{[}\log p(\mathcal{T}_{k}|\psi_{k})\big{]}-D_{\mathrm{KL}}(q_{\varphi_{k}}(\psi_{k})||p(\psi_{k}))\Big{]}.$$

$$\min_{\{\varphi_{k}\}_{k=1}^{K}}\frac{1}{K}\sum_{k=1}^{K}\Big{[}\mathbb{E}_{\psi_{k}\sim q_{\varphi_{k}}}\big{[}-\log p(\mathcal{T}_{k}|\psi_{k})\big{]}+D_{\mathrm{KL}}(q_{\varphi_{k}}(\psi_{k})||p(\psi_{k}))\Big{]}.$$

$$\min_{\Psi}\frac{1}{K}\sum_{k=1}^{K}\Big{[}\mathbb{E}_{\psi_{k}\sim q_{\Psi(\cdot)}}\big{[}-\log p(\mathcal{T}_{k}|\psi_{k})\big{]}+D_{\mathrm{KL}}(q_{\Psi(\cdot)}(\psi_{k})||p(\psi_{k}))\Big{]}.$$

$${\Theta }_0\leftarrow \theta ,$$

$${\Theta }_m\leftarrow {\Theta }_{m-1}-{\alpha }_m{\nabla }_{\Theta }{\mathcal{L}}_m^{(tr)}={\Theta }_{m-1}-{\Psi }_{\alpha }(\tau ,{\nabla }_{\Theta }{\mathcal{L}}_m^{(tr)}){\nabla }_{\Theta }{\mathcal{L}}_m^{(tr)},$$

$$\mathcal{L}^{(tr)}_{m}=L_{ce}\big{(}F(x^{(tr)};\Theta_{m-1}),y^{(tr)}\big{)},$$

$$z_m=F(x^{(te)};{\Theta }_m).$$

$${\hat{y}}_m^{(te)}\leftarrow v_m{z}_m+{\hat{y}}_{m-1}^{(te)}={\Psi }_v(\tau ,{\nabla }_{\Theta }{\mathcal{L}}_m^{(tr)})F(x^{(te)};{\Theta }_m)+{\hat{y}}_{m-1}^{(te)}.$$

$$\Delta {\alpha }_m,\Delta v_m={\text{FC}}_m(\text{concat}[\bar{\tau };\overline{{\nabla }_{{\Theta }_m}{\mathcal{L}}_m^{(tr)}}]),\text{or}$$

$$[\Delta {\alpha }_m,\Delta v_m],h_m=\text{LSTM}(\text{concat}[\bar{\tau };\overline{{\nabla }_{{\Theta }_m}{\mathcal{L}}_m^{(tr)}}],h_{m-1}),$$

$${\alpha }_m={\lambda }_1{\alpha }_m^{\prime }+(1-{\lambda }_1)\Delta \alpha ,v_m={\lambda }_2{v}_m^{\prime }+(1-{\lambda }_2)\Delta v,$$

$${\mathcal{L}}^{(te)}=L_{ce}({\hat{y}}_M^{(te)},y^{(te)}).$$

$${\Psi }_{\alpha }\leftarrow {\Psi }_{\alpha }-{\beta }_1{\nabla }_{{\Psi }_{\alpha }}{\mathcal{L}}^{(te)},{\Psi }_v\leftarrow {\Psi }_v-{\beta }_2{\nabla }_{{\Psi }_v}{\mathcal{L}}^{(te)},$$

Full text

1 Introduction

The ability of learning new concepts from a handful of examples is well-handled by humans, while in contrast, it remains challenging for machine models whose typical training requires a significant amount of data for good performance [34]. However, in many real-world applications, we have to face the situations of lacking a significant amount of training data, as e.g., in the medical domain. It is thus desirable to improve machine learning models to handle few-shot settings where each new concept has very scarce examples [39, 13, 70, 30].

Meta-learning methods aim to tackle the few-shot learning problem by transferring experience from similar few-shot tasks [7]. There are different meta strategies, among which the gradient descent based methods are particularly promising for today’s neural networks [13, 14, 20, 15, 38, 86, 70, 1, 84, 25, 83, 81, 74]. These methods follow a unified meta-learning procedure that contains two loops. The inner loop learns a base-learner for each individual task, and the outer loop uses the validation loss of the base-learner to optimize a meta-learner. In previous works [13, 14, 1, 70], the task of the meta-learner is to initialize the base-learner for the fast and efficient adaptation to the few training samples in the new task.

In this work, we aim to address two shortcomings of the previous works. First, the learning process of a base-learner for few-shot tasks is quite unstable [1], and often results in high-variance or low-confidence predictions. An intuitive solution is to train an ensemble of models and use the combined prediction which should be more robust [6, 54, 29]. However, it is not obvious how to obtain and combine multiple base-learners given the fact that a very limited number of training examples are available. Rather than learning multiple independent base-learners [79], we propose a novel method of utilizing the sequence of epoch-wise base-learners (while training a single base-learner) as the ensemble. Second, it is well-known that the values of hyperparameters, e.g., for initializing and updating models, are critical for best performance, and are particularly important for few-shot learning. In order to explore the optimal hyperparameters, we propose to employ the empirical Bayes method in the paradigm of meta-learning. In specific, we meta-learn hyperprior learners with meta-training tasks, and use them to generate task-specific hyperparameters, e.g., for updating and ensembling multiple base-learners. We call the resulting novel approach E3BM, which learns the Ensemble of Epoch-wise Empirical Bayes Models for each few-shot task. Our “epoch-wise models” are different models since each one of them is resulted from a specific training epoch and is trained with a specific set of hyperparameter values. During test, E3BM combines the ensemble of models’ predictions with soft ensembling weights to produce more robust results. In this paper, we argue that during model adaptation to the few-shot tasks, the most active adapting behaviors actually happen in the early epochs, and then converge to and even overfit to the training data in later epochs. Related works use the single base-learner obtained from the last epoch, so their meta-learners learn only partial adaptation experience [13, 70, 14, 25]. In contrast, our E3BM leverages an ensemble modeling strategy that adapts base-learners at different epochs and each of them has task-specific hyperparameters for updating and ensembling. It thus obtains the optimized combinational adaptation experience. Figure 1 presents the conceptual illustration of E3BM, compared to those of the classical method MAML [13] and the state-of-the-art SIB [25].

Our main contributions are three-fold. (1) A novel few-shot learning approach E3BM that learns to learn and combine an ensemble of epoch-wise Bayes models for more robust few-shot learning. (2) Novel hyperprior learners in E3BM to generate the task-specific hyperparameters for learning and combining epoch-wise Bayes models. In particular, we introduce four kinds of hyperprior learner by considering inductive [13, 70] and transductive learning methods [25], and each with either epoch-dependent (e.g., LSTM) or epoch-independent (e.g., epoch-wise FC layer) architectures. (3) Extensive experiments on three challenging few-shot benchmarks, miniImageNet [73], tieredImageNet [58] and Fewshot-CIFAR100 (FC100) [53]. We plug-in our E3BM to the state-of-the-art few-shot learning methods [13, 70, 25] and obtain consistent performance boosts. We conduct extensive model comparison and observe that our E3BM employing an epoch-dependent transductive hyperprior learner achieves the top performance on all benchmarks.

2 Related Works

Few-shot learning & meta-learning. Research literature on few-shot learning paradigms exhibits a high diversity from using data augmentation techniques [75, 77, 9] over sharing feature representation [2, 76] to meta-learning [18, 72]. In this paper, we focus on the meta-learning paradigm that leverages few-shot learning experiences from similar tasks based on the episodic formulation (see Section 3). Related works can be roughly divided into three categories. (1) Metric learning methods [73, 64, 71, 40, 78, 41, 24, 12, 82] aim to learn a similarity space, in which the learning should be efficient for few-shot examples. The metrics include Euclidean distance [64], cosine distance [73, 8], relation module [71, 41, 24] and graph-based similarity [62, 45]. Metric-based task-specific feature representation learning has also been presented in many related works [41, 78, 24, 12]. (2) Memory network methods [52, 53, 50] aim to learn training “experience” from the seen tasks and then aim to generalize to the learning of the unseen ones. A model with external memory storage is designed specifically for fast learning in a few iterations, e.g., Meta Networks [52], Neural Attentive Learner (SNAIL) [50], and Task Dependent Adaptive Metric (TADAM) [53]. (3) Gradient descent based methods [13, 14, 1, 57, 38, 20, 86, 70, 37, 25, 43] usually employ a meta-learner that learns to fast adapt an NN base-learner to a new task within a few optimization steps. For example, Rusu et al. [61] introduced a classifier generator as the meta-learner, which outputs parameters for each specific task. Lee et al. [37] presented a meta-learning approach with convex base-learners for few-shot tasks. Finn et al. [13] designed a meta-learner called MAML, which learns to effectively initialize the parameters of an NN base-learner for a new task. Sun et al. [70, 69] introduced an efficient knowledge transfer operator on deeper neural networks and achieved a significant improvement for few-shot learning models. Hu et al. [25] proposed to update base-learner with synthetic gradients generated by a variational posterior conditional on unlabeled data. Our approach is closely related to gradient descent based methods [13, 1, 70, 70, 69, 25]. An important difference is that we learn how to combine an ensemble of epoch-wise base-learners and how to generate efficient hyperparameters for base-learners, while other methods such as MAML [13], MAML++ [1], LEO [61], MTL [70, 69], and SIB [25] use a single base-learner.

Hyperparameter optimization. Building a model for a new task is a process of exploration-exploitation. Exploring suitable architectures and hyperparameters are important before training. Traditional methods are model-free, e.g., based on grid search [4, 42, 28]. They require multiple full training trials and are thus costly. Model-based hyperparameter optimization methods are adaptive but sophisticated, e.g., using random forests [27], Gaussian processes [65] and input warped Gaussian processes [67] or scalable Bayesian optimization [66]. In our approach, we meta-learn a hyperprior learner to output optimal hyperparameters by gradient descent, without additional manual labor. Related methods using gradient descent mostly work for single model learning in an inductive way [3, 10, 48, 47, 15, 49, 44, 46]. While, our hyperprior learner generates a sequence of hyperparameters for multiple models, in either the inductive or the transductive learning manner.

Ensemble modeling. It is a strategy [26, 85] to use multiple algorithms to improve machine learning performance, and which is proved to be effective to reduce the problems related to overfitting [35, 68]. Mitchell et al. [51] provided a theoretical explanation for it. Boosting is one classical way to build an ensemble, e.g., AdaBoost [16] and Gradient Tree Boosting [17]. Stacking combines multiple models by learning a combiner and it applies to both tasks in supervised learning [6, 54, 29] and unsupervised learning [63]. Bootstrap aggregating (i.e., Bagging) builds an ensemble of models through parallel training [6], e.g., random forests [22]. The ensemble can also be built on a temporal sequence of models [36]. Some recent works have applied ensemble modeling to few-shot learning. Yoon et al. proposed Bayesian MAML (BMAML) that trains multiple instances of base-model to reduce mete-level overfitting [80]. The most recent work [11] encourages multiple networks to cooperate while keeping predictive diversity. Its networks are trained with carefully-designed penalty functions, different from our automated method using empirical Bayes. Besides, its method needs to train much more network parameters than ours. Detailed comparisons are given in the experiment section.

3 Preliminary

In this section, we introduce the unified episodic formulation of few-shot learning, following [73, 57, 13]. This formulation was proposed for few-shot classification first in [73]. Its problem definition is different from traditional classification in three aspects: (1) the main phases are not training and test but meta-training and meta-test, each of which includes training and test; (2) the samples in meta-training and meta-testing are not datapoints but episodes, i.e. few-shot classification tasks; and (3) the objective is not classifying unseen datapoints but to fast adapt the meta-learned knowledge to the learning of new tasks.

Given a dataset $\mathcal{D}$ for meta-training, we first sample few-shot episodes (tasks) $\{\mathcal{T}\}$ from a task distribution $p(\mathcal{T})$ such that each episode $\mathcal{T}$ contains a few samples of a few classes, e.g., $5$ classes and $1$ shot per class. Each episode $\mathcal{T}$ includes a training split $\mathcal{T}^{(tr)}$ to optimize a specific base-learner, and a test split $\mathcal{T}^{(te)}$ to compute a generalization loss to optimize a global meta-learner. For meta-test, given an unseen dataset $\mathcal{D}_{un}$ (i.e., samples are from unseen classes), we sample a test task $\mathcal{T}_{un}$ to have the same-size training/test splits. We first initiate a new model with meta-learned network parameters (output from our hyperprior learner), then train this model on the training split $\mathcal{T}^{(tr)}_{un}$. We finally evaluate the performance on the test split $\mathcal{T}^{(te)}_{un}$. If we have multiple tasks, we report average accuracy as the final result.

4 An Ensemble of Epoch-wise Empirical Bayes Models

As shown in Fig. 2, E3BM trains a sequence of epoch-wise base-learners $\{\Theta_{m}\}$ with training data $\mathcal{T}^{(tr)}$ and learns to combine their predictions $\{z^{(te)}_{m}\}$ on test data $x^{(te)}$ for the best performance. This ensembling strategy achieves more robustness during prediction. The hyperparameters of each base-learner, i.e., learning rates $\alpha$ and combination weights $v$, are generated by the hyperprior learners conditional on task-specific data, e.g., $x^{(tr)}$ and $x^{(te)}$. This approach encourages the high diversity and informativeness of the ensembling models.

4.1 Empirical Bayes method

Our approach can be formulated as an empirical Bayes method that learns two levels of models for a few-shot task. The first level has hyperprior learners that generate hyperparameters for updating and combining the second-level models. More specifically, these second-level models are trained with the loss derived from the combination of their predictions on training data. After that, their loss of test data are used to optimize the hyperprior learners. This process is also called meta update, see the dashed arrows in Fig. 2.

In specific, we sample $K$ episodes $\{\mathcal{T}_{k}\}_{k=1}^{K}$ from the meta-training data $\mathcal{D}$. Let $\Theta$ denote base-learner and $\psi$ represent its hyperparameters. An episode $\mathcal{T}_{k}$ aims to train $\Theta$ to recognize different concepts, so we consider to use concepts related (task specific) data for customizing the $\Theta$ through a hyperprior $p(\psi_{k})$. To achieve this, we first formulate the empirical Bayes method with marginal likelihood according to hierarchical structure among data as follows,

[TABLE]

Then, we use variational inference [23] to estimate $\{p(\psi_{k})\}_{k=1}^{K}$. We parametrize distribution $q_{\varphi_{k}}(\psi_{k})$ with $\varphi_{k}$ for each $p(\psi_{k})$, and update $\varphi_{k}$ to increase the similarity betweeen $q_{\varphi_{k}}(\psi_{k})$ and $p(\psi_{k})$. As in standard probabilistic modeling, we derive an evidence lower bound on the log version of Eq. (1) to update $\varphi_{k}$,

[TABLE]

Therefore, the problem of using $q_{\varphi_{k}}(\psi_{k})$ to approach to the best estimation of $p(\psi_{k})$ becomes equivalent to the objective of maximizing the evidence lower bound [5, 23, 25] in Eq. (2), with respect to $\{\varphi_{k}\}_{k=1}^{K}$, as follows,

[TABLE]

To improve the robustness of few-shot models, existing methods sample a significant amount number of episodes during meta-training [13, 70]. Each episode employing its own hyperprior $p(\psi_{k})$ causes a huge computation burden, making it difficult to solve the aforementioned optimization problem. To tackle this, we leverage a technique called “amortized variational inference” [32, 59, 25]. We parameterize the KL term in $\{\varphi_{k}\}_{k=1}^{K}$ (see Eq. (3)) with a unified deep neural network $\Psi(\cdot)$ taking $x^{(tr)}_{k}$ (inductive learning) or $\{x^{(tr)}_{k},x^{(te)}_{k}\}$ (transductive learning) as inputs, where $x^{(tr)}_{k}$ and $x^{(te)}_{k}$ respectively denote the training and test samples in the $k$-th episode. In this paper, we call $\Psi(\cdot)$ hyperprior learner. As shown in Fig. 3, we additionally feed the hyperprior learner with the training gradients $\nabla\mathcal{L}_{\Theta}(\mathcal{T}^{(tr)}_{k})$ to $\Psi(\cdot)$ to encourage it to “consider” the current state of the training epoch. We mentioned in Sec. 1 that base-learners at different epochs are adapted differently, so we expect the corresponding hyperprior learner to “observe” and “utilize” this information to produce effective hyperparameters. By replacing $q_{\varphi_{k}}$ with $q_{\Psi(\cdot)}$, Problem (3) can be rewritten as:

[TABLE]

Then, we solve Problem (4) by optimizing $\Psi(\cdot)$ with the meta gradient descent method used in classical meta-learning paradigms [13, 70, 25]. We elaborate the details of learning $\{\Theta_{m}\}$ and meta-learning $\Psi(\cdot)$ in the following sections.

4.2 Learning the ensemble of base-learners

Previous works have shown that training multiple instances of the base-learner is helpful to achieve robust few-shot learning [79, 12]. However, they suffer from the computational burden of optimizing multiple copies of neural networks in parallel, and are not easy to generalize to deeper neural architectures. If include the computation of second-order derivatives in meta gradient descent [13], this burden becomes more unaffordable. In contrast, our approach is free from this problem, because it is built on top of optimization-based meta-learning models, e.g., MAML [13], MTL [70], and SIB [25], which naturally produce a sequence of models along the training epochs in each episode.

Given an episode $\mathcal{T}=\{\mathcal{T}^{(tr)},\mathcal{T}^{(te)}\}=\{\{x^{(tr)},y^{(tr)}\},\{x^{(te)},y^{(te)}\}\}$, let $\Theta_{m}$ denote the parameters of the base-learner working at epoch $m$ (w.r.t. $m$-th base-learner or BL-$m$), with $m\in\{1,...,M\}$. Basically, we initiate BL-$1$ with parameters $\theta$ (network weights and bias) and hyperparameters (e.g., learning rate $\alpha$), where $\theta$ is meta-optimized as in MAML [13], and $\alpha$ is generated by the proposed hyperprior learner $\Psi_{\alpha}$. We then adapt BL-$1$ with normal gradient descent on the training set $\mathcal{T}^{(tr)}$, and use the adapted weights and bias to initialize BL-$2$. The general process is thus as follows,

[TABLE]

[TABLE]

where $\alpha_{m}$ is the learning rate outputted from $\Psi_{\alpha}$, and $\nabla_{\Theta}\mathcal{L}^{(tr)}_{m}$ are the derivatives of the training loss, i.e, gradients. $\tau$ represents either $x^{(tr)}$ in the inductive setting, or $\{x^{(tr)},x^{(te)}\}$ in the transductive setting. Note that $\Theta_{0}$ is introduced to make the notation consistent, and a subscript $m$ is omitted from $\Psi_{\alpha}$ for conciseness. Let $F(x;\Theta_{m})$ denote the prediction scores of input $x$, so the base-training loss \mathcal{T}^{(tr)}=\big{\{}x^{(tr)},y^{(tr)}\big{\}} can be unfolded as,

[TABLE]

where $L_{ce}$ is the softmax cross entropy loss. During episode test, each base-learner BL-$m$ infers the prediction scores $z_{m}$ for test samples $x^{(te)}$,

[TABLE]

Assume the hyperprior learner $\Psi_{v}$ generates the combination weight $v_{m}$ for BL-$m$. The final prediction score is initialized as $\hat{y}^{(te)}_{1}=v_{1}z_{1}$ . For the $m$-th base epoch, the prediction $z_{m}$ will be calculated and added to $\hat{y}^{(te)}$ as follows,

[TABLE]

In this way, we can update prediction scores without storing base-learners or feature maps in the memory.

4.3 Meta-learning the hyperprior learners

As presented in Fig. 3, we introduce two architectures, i.e., LSTM or individual FC layers, for the hyperprior learner. FC layers at different epochs are independent. Using LSTM to “connect” all epochs is expected to “grasp” more task-specific information from the overall training states of the task. In the following, we elaborate the meta-learning details for both designs.

Assume before the $k$-th episode, we have meta-learned the base learning rates $\{\alpha_{m}^{\prime}\}_{m=1}^{M}$ and combination weights $\{v_{m}^{\prime}\}_{m=1}^{M}$. Next in the $k$-th episode, specifically at the $m$-th epoch as shown in Fig. 3, we compute the mean values of $\tau$ and $\nabla_{\Theta_{m}}\mathcal{L}^{(tr)}_{m}$, respectively, over all samples222In the inductive setting, training images are used to compute $\bar{\tau}$; while in the transductive setting, test images are additionally used.. We then input the concatenated value to FC or LSTM mapping function as follows,

[TABLE]

[TABLE]

where $h_{m}$ and $h_{m-1}$ are the hidden states at epoch $m$ and epoch $m-1$, respectively. We then use the output values to update hyperparameters as,

[TABLE]

where $\lambda_{1}$ and $\lambda_{2}$ are fixed fractions in $(0,1)$. Using learning rate $\alpha_{m}$, we update BL-$(m-1)$ to be BL-$m$ with Eq. (6). After $M$ epochs, we obtain the combination of predictions $\hat{y}^{(te)}_{M}$ (see Eq. (9)) on test samples. In training tasks, we compute the test loss as,

[TABLE]

We use this loss to calculate meta gradients to update $\Psi$ as follows,

[TABLE]

where $\beta_{1}$ and $\beta_{2}$ are meta-learning rates that determine the respective stepsizes for updating $\Psi_{\alpha}$ and $\Psi_{v}$. These updates are to back-propagate the test gradients till the input layer, through unrolling all base training gradients of $\Theta_{1}\sim\Theta_{M}$. The process thus involves a gradient through a gradient [13, 14, 70]. Computationally, it requires an additional backward pass through $\mathcal{L}^{(tr)}$ to compute Hessian-vector products, which is supported by standard numerical computation libraries such as TensorFlow [19] and PyTorch [55].

4.4 Plugging-in E3BM to baseline methods

The optimization of $\Psi$ relies on meta gradient descent method which was first applied to few-shot learning in MAML [13]. Recently, MTL [70] showed more efficiency by implementing that method on deeper pre-trained CNNs (e.g., ResNet-12 [70], and ResNet-25 [69]). SIB [25] was built on even deeper and wider networks (WRN-28-10), and it achieved top performance by synthesizing gradients in transductive learning. These three methods are all optimization-based, and use the single base-learner of the last base-training epoch. In the following, we describe how to learn and combine multiple base-learners in MTL, SIB and MAML, respectively, using our E3BM approach.

According to [70, 25], we pre-train the feature extractor $f$ on a many-shot classification task using the whole set of $\mathcal{D}$. The meta-learner in MTL is called scaling and shifting weights $\Phi_{SS}$, and in SIB is called synthetic information bottleneck network $\phi(\lambda,\xi)$. Besides, there is a common meta-learner called base-learner initializer $\theta$, i.e., the same $\theta$ in Fig. 2, in both methods. In MAML, the only base-learner is $\theta$ and there is no pre-training for its feature extractor $f$.

Given an episode $\mathcal{T}$, we feed training images $x^{(tr)}$ and test images $x^{(te)}$ to the feature extractor $f\odot\Phi_{SS}$ in MTL ($f$ in SIB and MAML), and obtain the embedding $e^{(tr)}$ and $e^{(te)}$, respectively. Then in MTL, we use $e^{(tr)}$ with labels to train base-learner $\Theta$ for $M$ times to get $\{\Theta_{m}\}_{m=1}^{M}$ with Eq. (6). In SIB, we use its multilayer perceptron (MLP) net to synthesize gradients conditional on $e^{(te)}$ to indirectly update $\{\Theta_{m}\}_{m=1}^{M}$. During these updates, our hyperprior learner $\Psi_{\alpha}$ derives the learning rates for all epochs. In episode test, we feed $e^{(te)}$ to $\{\Theta_{m}\}_{m=1}^{M}$ and get the combined prediction $\{z_{m}\}_{m=1}^{M}$ with Eq. (9). Finally, we compute the test loss to meta-update $[\Psi_{\alpha};\Psi_{v};\Phi_{SS};\theta]$ in MTL, $[\Psi_{\alpha};\Psi_{v};\phi(\lambda,\xi);\theta]$ in SIB, and $[f;\theta]$ in MAML. We call the resulting methods MTL+E3BM, SIB+E3BM, and MAML+E3BM, respectively, and demonstrate their improved efficiency over baseline models [70, 25, 13] in experiments.

5 Experiments

We evaluate our approach in terms of its overall performance and the effects of its two components, i.e. ensembling epoch-wise models and meta-learning hyperprior learners. In the following sections, we introduce the datasets and implementation details, compare our best results to the state-of-the-art, and conduct an ablation study.

5.1 Datasets and implementation details

Datasets. We conduct few-shot image classification experiments on three benchmarks: miniImageNet [73], tieredImageNet [58] and FC100 [53]. miniImageNet is the most widely used in related works [13, 25, 24, 70, 71, 25]. tieredImageNet and FC100 are either with a larger scale or a more challenging setting with lower image resolution, and have stricter training-test splits.

mini****ImageNet was proposed in [73] based on ImageNet [60]. There are $100$ classes with $600$ samples per class. Classes are divided into $64$, $16$, and $20$ classes respectively for sampling tasks for meta-training, meta-validation and meta-test. tiered****ImageNet was proposed in [58]. It contains a larger subset of ImageNet [60] with $608$ classes ($779,165$ images) grouped into $34$ super-class nodes. These nodes are partitioned into $20$, $6$, and $8$ disjoint sets respectively for meta-training, meta-validation and meta-test. Its super-class based training-test split results in a more challenging and realistic regime with test tasks that are less similar to training tasks. FC100 is based on the CIFAR100 [33]. The few-shot task splits were proposed in [53]. It contains $100$ object classes and each class has $600$ samples of $32\times 32$ color images per class. On these datasets, we consider the ($5$-class, $1$-way) and ($5$-class, $5$-way) classification tasks. We use the same task sampling strategy as in related works [13, 1, 25].

Backbone architectures. In MAML+E3BM, we use a $4$-layer convolution network (4CONV) [13, 1]. In MTL+E3BM, we use a $25$-layer residual network (ResNet-25) [56, 78, 69]. Followed by convolution layers, we apply an average pooling layer and a fully-connected layer. In SIB+E3BM, we use a $28$-layer wide residual network (WRN-28-10) as SIB [25].

The configuration of base-learners. In MTL [70] and SIB [25], the base-learner is a single fully-connected layer. In MAML [13], the base-learner is the $4$-layer convolution network. In MTL and MAML, the base-learner is randomly initialized and updated during meta-learning. In SIB, the base-learner is initialized with the averaged image features of each class. The number of base-learners $M$ in MTL+E3BM and SIB+E3BM are respectively $100$ and $3$, i.e., the original numbers of training epochs in [70, 25].

The configuration of hyperprior learners. In Fig. 3, we show two options for hyperprior learners (i.e., $\Psi_{\alpha}$ and $\Psi_{v}$). Fig. 3(a) is the epoch-independent option, where each epoch has two FC layers to produce $\alpha$ and $v$ respectively. Fig. 3(b) is the epoch-dependent option which uses an LSTM to generate $\alpha$ and $v$ at all epochs. In terms of the learning hyperprior learners, we have two settings: inductive learning denoted as “Ind.”, and transductive learning as “Tra.”. “Ind.” is the supervised learning in classical few-shot learning methods [13, 70, 37, 73, 64]. “Tra.” is semi-supervised learning, based on the assumption that all test images of the episode are available. It has been applied to many recent works [45, 24, 25].

Ablation settings. We conduct a careful ablative study for two components, i.e., “ensembling multiple base-learners” and “meta-learning hyperprior learners”. We show their effects indirectly by comparing our results to those of using arbitrary constant or learned values of $v$ and $\alpha$. In terms of $v$, we have $5$ ablation options: (v1) “E3BM” is our method generating $v$ from $\Psi_{v}$; (v2) “learnable” is to set $v$ to be update by meta gradient descent same as $\theta$ in [13]; (v3) “optimal” means using the values learned by option (a2) and freezing them during the actual learning; (v4) “equal” is an simple baseline using equal weights; (v5) “last-epoch” uses only the last-epoch base-learner, i.e., $v$ is set to $[0,0,...,1]$. In the experiments of (v1)-(v5), we simply set $\alpha$ as in the following (a4) [13, 70, 25]. In terms of $\alpha$, we have $4$ ablation options: (a1) “E3BM” is our method generating $\alpha$ from $\Psi_{\alpha}$; (a2) “learnable” is to set $\alpha$ to be update by meta gradient descent same as $\theta$ in [13]; (a3) “optimal” means using the values learned by option (a2) and freezing them during the actual learning; (a4) “fixed” is a simple baseline that uses manually chosen $\alpha$ following [13, 70, 25]. In the experiments of (a1)-(a4), we simply set $v$ as in (v5), same with the baseline method [70].

5.2 Results and analyses

In Table 1, we compare our best results to the state-of-the-arts. In Table 2, we present the results of using different kinds of hyperprior learner, i.e., regarding two architectures (FC and LSTM) and two learning strategies (inductive and transductive). In Fig. 4(a)(b), we show the validation results of our ablative methods, and demonstrate the change during meta-training iterations. In Fig. 4(c)(d), we plot the generated values of $v$ and $\alpha$ during meta-training.

Comparing to the state-of-the-arts. Table 1 shows that the proposed E3BM achieves the best few-shot classification performance in both $1$-shot and $5$-shot settings, on three benchmarks. Please note that [12] reports the results of using different backbones and input image sizes. We choose its results under the same setting of ours, i.e., using WRN-28-10 networks and $80\times 80\times 3$ images, for fair comparison. In our approach, plugging-in E3BM to the state-of-the-art model SIB achieves $1.6\%$ of improvement on average, based on the identical network architecture. This improvement is significantly larger as $2.9\%$ when taking MAML as baseline. All these show to be more impressive if considering the tiny overheads from pluging-in. For example, using E3BM adds only $0.04\%$ learning parameters to the original SIB model, and it gains only $5.2\%$ average overhead regarding the computational time. It is worth mentioning that the amount of learnable parameters in SIB+E3BM is around $80\%$ less than that of model in [12] which ensembles $5$ deep networks in parallel (and later learns a distillation network).

Hyperprior learners. In Table 2, we can see that using transductive learning clearly outperforms inductive learning, e.g., No. 5 vs. No. 4. This is because the “transduction” leverages additional data, i.e., the episode-test images (no labels), during the base-training. In terms of the network architecture, we observe that LSTM-based learners are slightly better than FC-based (e.g., No. 3 vs. No. 2). LSTM is a sequential model and is indeed able to “observe” more patterns from the adaptation behaviors of models at adjacent epochs.

Ablation study. Fig. 4(a) shows the comparisons among $\alpha$ related ablation models. Our E3BM (orange) again performs the best, over the models of using any arbitrary $\alpha$ (red or light blue), as well as over the model with $\alpha$ optimized by the meta gradient descent (blue) [13]. Fig. 4(b) shows that our approach E3BM works consistently better than the ablation models related to $v$. We should emphasize that E3BM is clearly more efficient than the model trained with meta-learned $v$ (blue) through meta gradient descent [13]. This is because E3BM hyperprior learners generate empirical weights conditional on task-specific data. The LSTM-based learners can leverage even more task-specific information, i.e., the hidden states from previous epochs, to improve the efficiency.

The values of $\alpha$ and $v$ learned by E3BM. Fig. 4(c)(d) shows the values of $\alpha$ and $v$ during the meta-training iterations in our approach. Fig. 4(c) show the base-learners working at later training epochs (e.g., BL-100) tend to get smaller values of $\alpha$. This is actually similar to the common manual schedule, i.e. monotonically decreasing learning rates, of conventional large-scale network training [21]. The difference is that in our approach, this is “scheduled” in a total automated way by hyperprior learners. Another observation is that the highest learning rate is applied to BL-$1$. This actually encourages BL-$1$ to make an influence as significant as possible. It is very helpful to reduce meta gradient diminishing when unrolling and back-propagating gradients through many base-learning epochs (e.g., $100$ epochs in MTL). Fig. 4(d) shows that BL-$1$ working at the initial epoch has the lowest values of $v$. In other words, BL-$1$ is almost disabled in the prediction of episode test. Intriguingly, BL-$25$ instead of BL-$100$ gains the highest $v$ values. Our explanation is that during the base-learning, base-learners at latter epochs get more overfitted to the few training samples. Their functionality is thus suppressed. Note that our empirical results revealed that including the overfitted base-learners slightly improves the generalization capability of the approach.

6 Conclusions

We propose a novel E3BM approach that tackles the few-shot problem with an ensemble of epoch-wise base-learners that are trained and combined with task-specific hyperparameters. In specific, E3BM meta-learns the hyperprior learners to generate such hyperparameters conditional on the images as well as the training states for each episode. Its resulting model allows to make use of multiple base-learners for more robust predictions. It does not change the basic training paradigm of episodic few-shot learning, and is thus generic and easy to plug-and-play with existing methods. By applying E3BM to multiple baseline methods, e.g., MAML, MTL and SIB, we achieved top performance on three challenging few-shot image classification benchmarks, with little computation or parametrization overhead.

Acknowledgments. This research was supported by the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant. We thank all reviewers and area chairs for their constructive suggestions.

A E3BM algorithms

Algorithm 1 summarizes the meta-training (line 1-10) and meta-testing (line 11-16) procedures in our E3BM approach. For clarity, the base-learning steps within a single episode are moved to Algorithm 2.

B Results with confidence intervals

In Table S1, we supplement the few-shot classification accuracy (%) on miniImageNet, tieredImageNet, and FC100 ($5$-class) with confidence intervals.

C Supplementary figures

Supplementary to Fig. 4(a)(b). In Fig. S1, we supplement the meta-validation accuracies for the $1$-shot and $5$-shot cases on miniImageNet, tieredImageNet, and FC100 (Note that Fig. 4 already has the miniImageNet $1$-shot results).

Supplementary to Fig. 4(c)(d). On the miniImageNet, we supplement the plots of $\alpha$ and $v$ in the $5$-shot case in Fig. S2(a)(b). On the tieredImageNet, we show the plots of $\alpha$ and $v$ in Fig. S2(c)(d) and (e)(f), respectively for $1$-shot and $5$-shot cases. On the FC100, we show the plots of $\alpha$ and $v$ in Fig. S2(g)(h) and (i)(j), respectively for $1$-shot and $5$-shot cases. Each figure demonstrates the values of $\alpha$ (or $v$) generated by the model “MTL+E3BM” as in Table 1.

D Backbone architectures

4CONV consists of $4$ layers with $3\times 3$ convolutions and $32$ filters, followed by batch normalization (BN), a ReLU nonlinearity, and $2\times 2$ max-pooling.

ResNet-12 has $3$ residual blocks. Each block has $4$ convolution layers with $3\times 3$ kernels. The number of filters starts from $160$ and is doubled every next block. After a global average pooling layer, it gets a $640$-dim embedding. This architecture follows [37].

ResNet-25 has $3$ residual blocks after an initial convolution layer. Each block has $8$ convolution layers with $3\times 3$ kernels. The number of filters starts from $160$ and is doubled every next block. After a global average pooling layer, it gets a $640$-dim embedding. This architecture follows [78].

WRN-28-10 has its depth and width set to $28$ and $10$, respectively. After a global average pooling in the last layer of the backbone, it gets a $640$-dimensional embedding. For this backbone, we resize the input image to $80\times 80\times 3$ for a fair comparison with related methods [70, 25]. Other details are the same as those with ResNet-25 [78, 61].

E Implementation details

MTL+E3BM. The meta learning rates for the scaling and shifting weights $\Phi_{SS}$ and the base-learner initializer $\theta$ are set to $1\times 10^{-4}$ uniformly. The base learning rates $\{\alpha_{m}^{\prime}\}_{m=1}^{M}$ (Fig. 3) are initialized as $1\times 10^{-2}$ [70, 78]. We meta-train MTL+E3BM for $10,000$ iterations and use the model, which has the highest meta-validation accuracy, for meta-test.

SIB+E3BM. The meta learning rates for both SIB network $\phi(\lambda,\xi)$ and base-learner initializer $\theta$ are set to $1\times 10^{-3}$ uniformly. The base learning rates $\{\alpha_{m}^{\prime}\}_{m=1}^{M}$ (Fig. 3) are initialized as $1\times 10^{-3}$ [25]. We meta-train SIB+E3BM for $50,000$ iterations and use the model, which has the highest meta-validation accuracy, for meta-test.

MAML+E3BM. MAML only contains a model initializer $\theta$, and we set its meta-learning rate as $1\times 10^{-3}$ [13]. The base learning rates $\{\alpha_{m}^{\prime}\}_{m=1}^{M}$ (Fig. 3) are initialized as $1\times 10^{-3}$. We meta-train MAML+E3BM for $60,000$ iterations and use the model, which has the highest meta-validation accuracy, for meta-test.

Shared hyperparameters. The meta learning rates for $\Psi_{\alpha}$ and $\Psi_{v}$ are set to $1\times 10^{-6}$ uniformly. For initializing $\{v_{m}^{\prime}\}_{m=1}^{M}$ (Fig. 3), we have two options. One is each $v_{m}^{\prime}$ is initialized as $1$/(number of base-learners), and the other one is that $\{v_{m}^{\prime}\}_{m=1}^{M-1}$ are initialized as [math] and $v_{M}^{\prime}$ as $1$. In Eq. (12) in Sec. 4.3, $\lambda_{1}$ and $\lambda_{2}$ are set to $1\times 10^{-4}$. For the rest of the hyperparameters, we follow the original settings of baselines [13, 25, 70].

Constraints for $v$ and $\alpha$. In the constraint mode, we applied the constraints on $v$ and $\alpha$ to force them to be positive and smaller than $1$. We did not have any constraint for $\Delta v$ or $\Delta\alpha$. Please note that the constraints are not applied in the default setting.

Dataloader For MAML, we use the same dataloader as [13]. For MTL, we follow [78, 82, 81]. For SIB, we follow [25].

F Ablation results for MAML

In Table S2, we supplement the ablation results for “MAML+E3BM” on miniImageNet, tieredImageNet, and FC100 ($5$-class).

G The inference time and the number of parameters

In Table S3, we supplement the the inference time and the number of parameters of baselines ($100$ epochs, miniImageNet, $5$-way $1$-shot, on NVIDIA V100 GPU)

H Executing the source code with PyTorch

We provide our PyTorch code at https://gitlab.mpi-klsb.mpg.de/yaoyaoliu/e3bm. To run this repository, we kindly advise you to install python 3.6 and PyTorch 1.2.0 with Anaconda. You may download Anaconda and read the installation instruction on the official website (https://www.anaconda.com/download/).

Create a new environment and install PyTorch and torchvision on it:

⬇

1conda create --name e3bm-pytorch python=3.6

2conda activate e3bm-pytorch

3conda install pytorch=1.2.0

4conda install torchvision -c pytorch

Install other requirements:

⬇

1pip install -r requirements.txt

Run meta-training with default settings (data and pre-trained model will be downloaded automatically):

⬇

1python main.py -backbone resnet12 -shot 1 -way 5 -mode meta_train -dataset miniimagenet

2python main.py -backbone resnet12 -shot 5 -way 5 -mode meta_train -dataset miniimagenet

3python main.py -backbone resnet12 -shot 1 -way 5 -mode meta_train -dataset tieredimagenet

4python main.py -backbone resnet12 -shot 5 -way 5 -mode meta_train -dataset tieredimagenet

Run pre-training with default settings:

⬇

1python main.py -backbone resnet12 -mode pre_train -dataset miniimagenet

2python main.py -backbone resnet12 -mode pre_train -dataset tieredimagenet