Posterior-Guided Neural Architecture Search

TL;DR

This paper introduces a Bayesian approach to neural architecture search (NAS) that estimates the joint posterior of architectures and weights, enabling efficient sampling and better weight sharing insights, validated on image classification tasks.

Contribution

It proposes a posterior-guided NAS method using Variational Dropout for fully gradient-based posterior approximation, improving efficiency and understanding of weight sharing in NAS.

Findings

Achieves competitive accuracy on CIFAR-10 and CIFAR-100.

Searches architectures in 11 GPU days with good trade-offs.

Provides deeper insights into weight sharing in NAS.

Abstract

The emergence of neural architecture search (NAS) has greatly advanced the research on network design. Recent proposals such as gradient-based methods or one-shot approaches significantly boost the efficiency of NAS. In this paper, we formulate the NAS problem from a Bayesian perspective. We propose explicitly estimating the joint posterior distribution over pairs of network architecture and weights. Accordingly, a hybrid network representation is presented which enables us to leverage the Variational Dropout so that the approximation of the posterior distribution becomes fully gradient-based and highly efficient. A posterior-guided sampling method is then presented to sample architecture candidates and directly make evaluations. As a Bayesian approach, our posterior-guided NAS (PGNAS) avoids tuning a number of hyper-parameters and enables a very effective architecture sampling in posterior probability space. Interestingly, it also leads to a deeper insight into the weight sharing used in the one-shot NAS and naturally alleviates the mismatch between the sampled architecture and weights caused by the weight sharing. We validate our PGNAS method on the fundamental image classification task. Results on Cifar-10, Cifar-100 and ImageNet show that PGNAS achieves a good trade-off between precision and speed of search among NAS methods. For example, it takes 11 GPU days to search a very competitive architecture with 1.98% and 14.28% test errors on Cifar10 and Cifar100, respectively.

Tables

Table 1. Table 1: Performance comparison with other state-of-the-art results. Please note that we do not fine-tune the network searched by our method. ∗ indicates the architecture searched by sampling 10000 candidates. Full table is in supplementary material.

Method	Error(%)	GPUs Days	Params(M)	Search Method
shake-shake + cutout (?)	2.56	-	26.2	-
NAS + more filters (?)	3.65	22400	37.4	RL
NASNET-A + cutout (?)	2.65	1800	3.3	RL
PathLevel EAS + PyramidNet + cutout (?)	2.30	8.3	13.0	RL
ENAS + cutout (?)	2.89	0.5	4.6	RL
EAS (DenseNet) (?)	3.44	10	10.7	RL
AmoebaNet-A + cutout (?)	3.34	3150	3.2	evolution
Hierachical Evo (?)	3.63	300	61.3	evolution
PNAS (?)	3.63	225	3.2	SMBO
BayesNAS + PyramidNet(?) + cutout	2.40	0.1	3.4	gradient-based
DARTS + cutout (?)	2.83	4	3.4	gradient-based
SNAS + cutout (?)	2.85	1.5	2.8	gradient-based
NAONet + cutout (?)	2.07	200	128	gradient-based
One-Shot Top (?)	3.70	-	45.3	sampling-based
SMASH (?)	4.03	1.5	16.0	sampling-based
PGNAS-MI(ours)	2.06	6.5	33.4	guided sampling
PGNAS-MI∗(ours)	1.98	11.1	32.8	guided sampling

Table 2. Table 2: State-of-art results on Cifar-100. Please note that we do not fine-tune the network searched by our method. ”PGNAS-MI + constant L2 term” indicates we replace the second L2 term in Eq. 15 with conventional a L2 term with a constant weight.

Method	Error(%)	GPUs Days	Params(M)	Search Method
NASNET-A (?)	19.70	1800	3.3	RL
ENAS (?)	19.43	0.5	4.6	RL
AmoebaNet-B (?)	17.66	3150	2.8	evolution
PNAS (?)	19.53	150	3.2	SMBO
NAONet + cutout (?)	14.36	200	128	gradient-based
PGNAS-MI + constant L2 term(ours)	17.41	-	39.6	-
PGNAS-MI(ours)	14.28	11	46.4	guided sampling

Table 3. Table 3: Impact of the guided sampling. w/o FT and w/ FT indicate whether the searched one is fine-tuned on the dataset.

	SupNet-EI		SupNet-E		SupNet-MI		SupNet-M
	Err.	Param.	Err.	Param.	Err.	Param.	Err.	Param.
Full model	2.78%	15.3M	2.98%	4.6M	-%	72.7M	2.58%	26.2M
Random w/o FT	13.45%	10.7M	15.87%	3.0M	9.75%	35.4M	2.63%	22.4M
Random w/ FT	3.16%	10.7M	3.47%	3.0M	2.69%	35.4M	2.56%	22.4M
PGNAS	2.56%	10.8M	2.73%	3.1M	2.06%	33.4M	2.20%	21.6M

Table 4. Table 4: Impact of the weight prior l 2 superscript 𝑙 2 l^{2} on SupNet-EI.

$l^2$	50	150	250	500
Error(%)	2.13	2.06	2.27	2.39
Params(M)	49.9	33.4	23.8	18.2

Table 5. Table 5: Impact of the temperature τ 𝜏 \tau . † denotes fine-tuning.

	EI	M	EI†
$\tau =\frac{2}{3}$	2.74%	2.49%	2.68%
$\tau =\frac{1}{5}$	2.56%	2.20%	-

Table 6. Table 6: Impact of the number of candidates on SupNet-MI.

	0.05K	0.5k	1.5k	5.0k	10k	20k
Error(%)	2.17	2.06	2.06	2.04	1.98	-
$\Delta$GDays	0.02	0.23	0.69	2.31	4.63	9.26

Table 7. Table 7: Proportions of dropped channels and operations.

	MI∗	M	E	EI
Channel-level	0.44	0.18	0.33	0.29
Operation-level	0.26	0.10	0.19	0.21

Table 8. Table 8: Comparison with other state-of-the-art results on ImageNet. Please note our model is directly searched on ImageNet.

Method	Error(%)(Top1/Top5)	GPUs Days	Params(M)	Search Method
NASNET-A (?)	26.0/8.4	1800	5.3	RL
AmoebaNet-C (?)	24.3/7.6	3150	6.4	evolution
PNAS (?)	25.8/8.1	225	5.1	SMBO
BayesNAS (${\lambda }_w^o=0.005$) (?)	26.5/8.9	0.2	3.9	gradient-based
FBNet-C (?)	25.1/-	9	5.5	gradient-based
DARTS (?)	26.9/9.0	4	4.9	gradient-based
SNAS (?)	27.3/9.2	1.5	4.3	gradient-based
One-Shot Top (?)	26.2/-	-	6.8	sampling-based
SinglePath (?)	25.3/-	12	-	sampling-based
PGNAS-D-121(ours)	24.8/7.5	26	6.6	guided sampling

Table 9. Table 9: Test results on ImageNet with inflated ResNet-50.

Model	Res50	Inflated Res50	PGNAS-R-50
Error	23.96%	22.93%	22.73%
Params	25.6M	44.0M	26.0M

Equations

$$w_{\alpha }^{\ast }={\mathrm{argmin}}_{w_{\alpha }}\mathcal{L}(\mathcal{M}(\alpha ,w_{\alpha });{\mathcal{D}}_t),$$

$${\alpha }^{\ast }={\mathrm{argmin}}_{\alpha \in G}\mathcal{L}(\mathcal{M}(\alpha ,w_{\alpha }^{\ast });{\mathcal{D}}_v),$$

$$r^{\ast },w^{\ast }={\mathrm{argmin}}_{r,w}\mathcal{L}(\mathcal{M}(G_r,w);{\mathcal{D}}_t).$$

$$w_{{\alpha }_o}^{\ast }={\mathrm{argmin}}_{w_{{\alpha }_o}}\mathcal{L}(\mathcal{M}({\alpha }_o,w_{{\alpha }_o});{\mathcal{D}}_t).$$

$${\alpha }^{\ast }={\mathrm{argmin}}_{\alpha \subseteq {\alpha }_o}\mathcal{L}(\mathcal{M}(\alpha ,w_{\alpha }^{\ast });{\mathcal{D}}_v),$$

$${\alpha }^{\ast }={\mathrm{argmin}}_{\alpha ,w\sim p(\alpha ,w\mid {\mathcal{D}}_t)}\mathcal{L}(\mathcal{M}(\alpha ,w);{\mathcal{D}}_v).$$

$${\theta }^{\ast }={\mathrm{argmin}}_{\theta }L(q_{\theta }(\alpha ,w),p(\alpha ,w\mid {\mathcal{D}}_t)),$$

$$\begin{array}{r}p(\mathbf{\phi }\mid X,Y)=\frac{p(Y\mid X,\mathbf{\phi })p(\mathbf{\phi })}{{\int }_{\mathbf{\phi }}p(Y\mid X,\mathbf{\phi })},\end{array}$$

$$\begin{array}{rl}& {\theta }^{\ast }={\mathrm{argmin}}_{\theta }{L}_d(q_{\theta }(\mathbf{\phi }),p(\mathbf{\phi }\mid {\mathcal{D}}_t)),\\ & {\alpha }^{\ast }={\mathrm{argmin}}_{\phi \sim q_{{\theta }^{\ast }}(\mathbf{\phi })}\mathcal{L}(\phi ;D_v).\end{array}$$

$$\begin{array}{rl}{\mathcal{L}}_{VI}(\theta ):& =KL(q_{\theta }(\mathbf{\phi })\mid \mid p(\mathbf{\phi }))\\ & -\sum _{i=1}^N\int q_{\theta }(\mathbf{\phi })\log p(y_i\mid f^{\mathbf{\phi }}(x_i))d\mathbf{\phi },\end{array}$$

$$\begin{array}{r}{\mathbf{w}}_{l,k}^s=m_{l,k}^s\cdot {\mathbf{z}}_{l,k}^s,where{\mathbf{z}}_{l,k}^s\sim Bernoulli({\tilde{p}}_l^s),\end{array}$$

$$\begin{array}{r}{\mathbf{\phi }}_{l,k}^s=m_{l,k}^s\cdot {\mathbf{ϵ}}_{l,k}^s,where{\mathbf{ϵ}}_{l,k}^s={\mathbf{z}}_{l,k}^s\cdot {\mathbf{\alpha }}_{l,k}^s,\end{array}$$

$$\begin{array}{r}{\mathbf{\phi }}_{l,k}^s=m_{l,k}^s\cdot {\mathbf{ϵ}}_{l,k}^s,where{\mathbf{ϵ}}_{l,k}^s\sim Bernoulli({\tilde{p}}_l^s\cdot {\overline{p}}_l^s).\end{array}$$

$$\begin{array}{rl}{\mathcal{L}}_{MC}(\theta ):=& KL(q_{\theta }(\mathbf{\phi })\mid \mid p(\mathbf{\phi }))-\sum _{i=1}^N\log p(y_i\mid f^{\theta ,{\mathbf{ϵ}}_i}(x_i)),\\ & s.t.{\mathbb{E}}_{\mathbf{ϵ}}\{{\mathcal{L}}_{MC}(\theta )\}={\mathcal{L}}_{VI}(\theta ).\end{array}$$

$$\begin{array}{rl}\frac{\partial }{\partial p_l^s\partial m_{l,k}^s}[& -\frac{1}{N}(\sum _{i=1}^{N}p(y_i\mid f^{\theta ,{\mathbf{ϵ}}_i}(x_i))+\mathcal{H}(p_l^s))\\ & +\sum _{l,k,s}\frac{(d_{l,k}^s{)}^2(1-p_l^s)}{2N}\text{Arrowvert}{m}_{l,k}^s{\text{Arrowvert}}^2],\end{array}$$

$$\begin{array}{rl}{\mathbf{ϵ}}_{l,k}^s=\sigma (\frac{1}{\tau }[& \log p_l^s-\log (1-p_l^s)+\log (\log r_2)\\ & -\log (\log r_1)])\\ & s.t.r_1,r_2\sim Uniform(0,1),\end{array}$$

Full text

1 Introduction

Neural architecture search (NAS), which automates the design of artificial neural networks (ANN), has received increasing attention due to its ability of finding ANNs with similar or even better performance than manually designed ones. Essentially, NAS is a bi-level optimization problem. Given an neural architecture $\alpha$ which belongs to a pre-defined search space $G$, the lower-level objective optimizes the weight $w_{\alpha}$ of the architecture $\alpha$ as

[TABLE]

where $\mathcal{L}$ is a loss criterion that measures the performance of network $\mathcal{M}(\alpha,w_{\alpha})$ with architecture $\alpha$ and weight $w_{\alpha}$ on the training dataset $\mathcal{D}_{t}$; whereas the upper-level objective optimizes the network architecture $\alpha$ with the weight $w^{*}_{\alpha}$ that has been optimized by the lower-level task as

[TABLE]

on the validation dataset $\mathcal{D}_{v}$. To solve this bi-level problem, different approaches have been proposed, e.g. evolutionary-based methods (?; ?), reinforcement learning based schemes (?; ?; ?; ?; ?; ?; ?; ?; ?; ?; ?; ?) or gradient-based algorithms (?; ?; ?; ?). However, most of these methods suffer from high computational complexity (often in the orders of thousands of GPU days) (?) (?) (?) (?) (?), or lack of convergence guarantee (?; ?; ?).

Instead of directly tackling the bi-level optimization problem, some attempts (?; ?; ?; ?) relax the discrete search space $G$ to be continuous. Given one continuous relaxation $G_{r}$ of topology $r$, the weight and architecture can be jointly optimized by the single objective function

[TABLE]

Then the optimal architecture $\alpha^{*}$ is derived by discretizing the continuous one $G_{r^{*}}$. These methods greatly simplify the optimization problem and enable end-to-end training. However, since the validation set $\mathcal{D}_{v}$ is excluded in Eq. 3, the search results are usually biased on training datasets.

More recent NAS methods tend to reduce the computational complexity by decoupling the bi-level optimization problem into a sequential one (?; ?; ?). Specifically, the search space $G$ here is defined by an over-parameterized super-network (one-shot model) with architecture $\alpha_{o}$. Then the one-shot NAS starts with optimizing the weights $w_{\alpha_{o}}$ of the super-network $\alpha_{o}$ by Eq. 1, resulting $w_{\alpha_{o}}^{*}$ as

[TABLE]

After that, a number of sub-architectures $\alpha$ are sampled from $\alpha_{o}$. Then the best-performing one is selected by

[TABLE]

where $w^{*}_{\alpha}$ is inherited from $w^{*}_{\alpha_{o}}$. We notice that one core assumption in the one-shot NAS method is that the best-performing sub-network shares weights with the optimal super-network. Thus the sampled sub-networks do not need to be re-trained in the searching process. This assumption, on the one hand, greatly boosts the efficiency of NAS; on the other, it could lead to mismatches between weights and architectures of sampled sub-networks, which jeopardizes the following ranking results (?). More clues can be found that most one-shot methods rely on fine-tuning to further improve the performance of the found model. In addition, the sampling process in current one-shot NAS is much less explored which has a big impact on the performance as demonstrated later in our study (Table 3).

In this paper, we propose modeling the NAS problem from a Bayesian perspective and accordingly present a new NAS strategy, i.e. posterior-guided NAS (PGNAS). In PGNAS, given a search space $G$, we estimate the posterior distribution $p(\alpha,w\mid\mathcal{D}_{t})$ ($\alpha\in G$) over pairs of architecture and weight ($\alpha,w$). Then optimal architecture is searched by

[TABLE]

However, since the posterior distribution $p(\alpha,w\mid\mathcal{D}_{t})$ is intractable, we approximate it by a variational distribution $q_{\theta}(\alpha,w)$ as

[TABLE]

where $\theta$ denotes the variational parameters and $L$ measures the distance between two distributions. Still, finding $\theta^{*}$ is not trivial. Therefore, we further propose a hybrid network representation to facilitate an end-to-end trainable solution. In short, our PGNAS manages to leverage the training dataset to estimate the joint posterior distribution over network architecture and weight, based on which an efficient and effective sampling in posterior probability space is enabled.

We recently noticed that there is a parallel work (?) that employs the sparse Bayesian learning (SBL) (?) in NAS. This work focuses on tackling two problems in the one-shot NAS, i.e., neglect of dependencies between nodes and problematic magnitude-based operation pruning. It encodes the node dependency logic into the prior distribution of the architecture and exploit the SBL to obtain the most sparse solution for the prior distribution. The entropy of the derived prior distribution with a pre-defined threshold is then used as the criterion to prune operations. Differently, we directly formulate NAS in a posterior probability space, and sample pairs of architecture and weights in this space to search for good architectures.

In summary, the main contributions of this work are:

• •

We convert NAS to a distribution construction problem. We formulate the NAS problem by Bayesian theory and propose to estimate the joint posterior distribution of pairs of network architecture and weights. It thus enables a very efficient search process in the probability space through a posterior-guided sampling. The presented PGNAS achieves a good trade-off between performance and speed of search, e.g. it speeds up the search process by 20X compared with the second best while achieving the best performance with 1.98% test error on Cifar10.

• •

An end-to-end trainable solution is proposed to approximate the joint posterior distribution on architecture and weights. In particular, a hybrid network representation is presented to facilitate the Variational Dropout (?)-based solution, which makes the approximation fully gradient-based and highly efficient.

• •

Our PGNAS models and samples architecture and weights jointly. It thus reduces the mismatch between the architecture and weights caused by weight sharing in the one-shot NAS and improves the reliability of evaluation for the sampled network. In our PGNAS, the sampled weights can be adopted directly by the searched architecture to achieve high performance without fine-tuning.

• •

We find that the weight sharing in our PGNAS can be viewed as a re-parametrization that enables us to estimate the posterior distribution via end-to-end optimization. This finding may help in better understanding of weight sharing that is well-accepted in the one-shot NAS.

2 PGNAS

In this section, we first formulate the target problem in our PGNAS via the joint posterior distribution. Then an end-to-end trainable solution is presented to approximate the posterior distribution. At last, an efficient sampling and ranking scheme is described to facilitate the search process.

In the following, we use $w_{l}^{s}\in R^{K_{l}\times H_{l}\times s\times s}$ to denote the convolution weight matrix in layer $l$ with spatial kernel size $s$, where $K_{l}$ and $H_{l}$ denote the number of input and output channels in layer $l$, respectively. $w_{l,k}^{s}\in R^{1\times H_{l}\times s\times s}$ denotes the sliced kernel operated on the $k^{th}$ ($1\leq k\leq K_{l}$) input channel dimension. $w=\{w_{l,k}^{s}\}$ presents weights of the whole super-network. As deriving an architecture $\alpha$ is equivalent to selecting a set of convolution kernels, we introduce a set of random variables $\boldsymbol{\alpha}=\{\boldsymbol{\alpha}_{l,k}^{s}\}$ ( $\boldsymbol{\alpha}_{l,k}^{s}\ \in\{0,1\}$) to indicates deactivating or activating convolution kernel $\boldsymbol{w}_{l,k}^{s}$ by setting the corresponding $\boldsymbol{\alpha}_{l,k}^{s}$ to 1 or 0, respectively. We use boldface for random variables later on.

2.1 Problem Formulation

As defined in Eq. 6 and 7, in order to search a good architecture from the posterior probability space, we need to construct a joint posterior distribution over $\boldsymbol{\alpha}$ and $\boldsymbol{w}$ which is usually very hard to obtain. However, we notice it is not necessary to compute it explicitly as deactivating or activating a convolution kernel is also equivalent to multiplying a binary mask to the kernel. Hence, we combine the two random variables into a new one $\boldsymbol{\varphi}=\{\mathcal{\varphi}_{l,k}^{s}\}$, where $\mathcal{\varphi}_{l,k}^{s}\ =\boldsymbol{w}_{l,k}^{s}\cdot\boldsymbol{\alpha}_{l,k}^{s}$. Now, the key problem in our PGNAS becomes the approximation of the posterior distribution over the hybrid network representation $\boldsymbol{\varphi}$. Mathematically,

[TABLE]

where $X=\{x_{i}midi=1,...,N\}$ and $Y=\{y_{i}\mid i=1,...,N\}$ denote $N$ training samples and labels, respectively. $p(Y\mid X,\boldsymbol{\varphi})$ is the likelihood that can be inferred by $\prod_{i=1}^{N}p(y_{i}\mid f^{\boldsymbol{\varphi}}(x_{i}))$, where $f^{\boldsymbol{\varphi}}$ denotes a sub-network defined by hybrid representation $\boldsymbol{\varphi}$. $p(\boldsymbol{\varphi})$ is the a priori distribution of hybrid representation $\boldsymbol{\varphi}$. Because the marginalized likelihood $\int_{\boldsymbol{\varphi}}p(Y\mid X,\boldsymbol{\varphi})$ in Eq. 8 is intractable, we use a variational distribution $q_{\theta}(\boldsymbol{\varphi})$ to approximate Eq. 8 and reformulate our target problem as

[TABLE]

Here we choose KL divergence and accuracy to instantiate $\mathcal{L}_{d}$ and $\mathcal{L}$, respectively.

2.2 Posterior Distribution Approximation

To solve Eq. 9, we employ Variational Inference(VI) which minimizes the negative Evidence Lower Bound (ELBO)

[TABLE]

Here we propose solving Eq. 10 by the network friendly Variational Dropout.

Approximation by Network Training

Due to the hybrid network representation, we can use the re-parametrization trick (?) as in (?; ?) to solve Eq. 10. We choose a deterministic and differentiable transformation function $g(\cdot,\cdot)$ that re-parameterizes the $q_{\theta}(\boldsymbol{\varphi})$ as $\boldsymbol{\varphi}=g(\theta,\boldsymbol{\epsilon})$, where $\boldsymbol{\epsilon}\sim p(\boldsymbol{\epsilon})$ is a parameter-free distribution. Take a uni-variate Gaussian distribution $x\sim q_{\theta}(x)=\mathcal{N}(\mu,\sigma)$ as an example, its re-parametrization can be $x=g(\theta,\boldsymbol{\epsilon})=\mu+\sigma\boldsymbol{\epsilon}$ with $\boldsymbol{\epsilon}\sim\mathcal{N}(0,1)$, where $\mu$ and $\sigma$ are the variational parameters $\theta$. Gal et.al. in (?; ?) show that when the network weight is re-parameterized with

[TABLE]

the function draw w.r.t. variational distribution over network weights $\boldsymbol{w}$ can be efficiently implemented via network inference. Concretely, the function draw is equivalent to randomly drawing masked deterministic weight matrix $m=\{m^{s}_{l,k}\}$ in neural networks, which is known as the Dropout operations (?). Similarly, we replace $\boldsymbol{w}_{l,k}^{s}$ in our hybrid representation $\boldsymbol{\varphi}_{l,k}^{s}=\boldsymbol{w}_{l,k}^{s}\cdot\boldsymbol{\alpha}_{l,k}^{s}$ with $m_{l,k}^{s}\cdot\boldsymbol{z}_{l,k}^{s}$, and reformulate $\boldsymbol{\varphi}_{l,k}^{s}$ as

[TABLE]

In Eq. 12, we have an additional random variable $\boldsymbol{\alpha}_{l,k}^{s}$ that controls the activation of kernels whose distribution is unknown. Here we propose using the marginal probability $p(\boldsymbol{\alpha}_{l,k}^{s}\mid X,Y)$ to characterize its behavior, because the marginal can reflect the expected probability of selecting kernel $\boldsymbol{\alpha}_{l,k}^{s}$ given the training dataset. It exactly matches the real behavior if the selections of kernels in a super-network are independent. Since the joint distribution of network architecture $\boldsymbol{\alpha}=\{\boldsymbol{\alpha}_{l,k}^{s}\}$ is a multivariate Bernoulli distribution, its marginal distribution obeys $Bernoulli(\overline{p}_{l,k}^{s})$ (?), where $\overline{p}_{l,k}^{s}$ is to be optimized. Therefore, we have

[TABLE]

Here we omit the subscript $k$ in the original $Bernoulli(\overline{p}_{l,k}^{s})$ because the importance of branches which come from the same kernel size group and layer should be identical. By replacing $\widetilde{p}_{l}^{s}\cdot\overline{p}_{l}^{s}$ with a new variable $p_{l}^{s}$, Eq. 13 has the same form as Eq. 11. Then, we can obtain

[TABLE]

where variational parameters $\theta=\{m_{l,k}^{s}\}$ are composed of the deterministic kernel weights. $\boldsymbol{\epsilon}_{i}=\{\boldsymbol{\epsilon}_{l,k}^{s}\}_{i}$ are the $i^{th}$ sampled random variables which encodes the distribution of network architecture. Eq. 14 indicates that the (negative) ELBO can be computed very efficiently. It is equivalent to the KL term minus the log likelihood that is inferenced by the super-network $f^{\boldsymbol{\varphi}}$ (now reparameterized as $f^{\theta,\boldsymbol{\epsilon}}$). During each network inference, convolution kernels are randomly deactivated w.r.t. probability $p=\{p_{l}^{s}\}$, which is exactly equivalent to a dropout neural network.

Now, approximating posterior distribution over the hybrid network representation is converted to optimizing the network $f^{\boldsymbol{\varphi}}$ with dropout and a KL regularization term. If the derivatives of both terms are tractable, we can efficiently optimize it in an end-to-end fashion.

Network Optimization

In addition to the variational parameters $\theta$, the variable $\widetilde{p}$ and $\overline{p}$ in Eq. 13 should also be optimized (either via grid-search (?) or gradient-based method (?)). So we need to compute $\frac{\partial}{\partial p\partial m}\mathcal{L}_{MC}(\theta)$. If each convolution kernel is deactivated with a prior probability $u^{s}_{l,k}$ along with a Gaussian weight prior $\mathcal{N}(\boldsymbol{w}_{l,k}^{s};0,I/(d_{l,k}^{s})^{2})$, then the priori distribution for the hybrid representation $\boldsymbol{\varphi}$ is exactly a spike and slab prior $p(\boldsymbol{\varphi}_{l,k}^{s})=u^{s}_{l,k}\cdot\delta(\boldsymbol{w}_{l,k}^{s}-0)+(1-u^{s}_{l,k})\cdot\mathcal{N}(\boldsymbol{w}_{l,k}^{s};0,I/(d_{l,k}^{s})^{2})$, where $d^{s}_{k,l}$ is prior length scale. Following (?; ?), the derivatives of Eq. 14 can be approximated as

[TABLE]

where $\mathcal{H}(p^{s}_{l})=\sum_{l,s}k_{l}^{s}\cdot p_{l}^{s}\cdot\log p_{l}^{s}$ and $k_{l}^{s}$ denotes the number of input channels for convolution kernel of spatial size $s$ at layer $l$. Please note that the above derivation is obtained by setting the prior $u$ to be zero, which indicates the network architecture prior is set to be the whole super-network. The motivation of employing $u=0$ is that a proper architecture prior is usually difficult to acquire or even estimate, but $u=0$ can be a reasonable one when we choose the over-parameterized network that proves effective on many tasks as our super-network. Besides, $u=0$ provides us a more stable way to optimize the $\mathcal{L}_{MC}(\theta)$ (?). So, we will use the super-network that are built upon manually designed networks in our experiments.

Intuitively, the derived Eq. 15 tries to find a distribution that can interpret the data well (the first likelihood term) and keep the architecture as sparse as possible (the second and third regularization terms). Please note here that the L2-like term in Eq. 15 is not a conventional L2 regularization term. Its coefficient $(d_{l,k}^{s})^{2}(1-p_{l}^{s})$ correlates with architecture selection parameter $p$ and thus encourages the selection of kernels whose learned weights $w$ are not only representative but also sparse (as the gradients w.r.t. $p$ here rely on the sparsity of learned $w$). It is consistent with our design where we correlate the network weights and architecture by proposing the hybrid network representation $\boldsymbol{\varphi}$.

Since the first term in Eq. 15 involves the derivatives of the non-differentiable Bernoulli distribution (remember $\boldsymbol{\epsilon}_{l,k}^{s}\sim Bernoulli({p}_{l}^{s})$ in Eq. 13), we thus employ the Gumbel-softmax (?) to relax the discrete distribution $Bernoulli(p_{l}^{s})$ to continuous space and the $\boldsymbol{\epsilon}$ in Eq. 15 and Eq. 13 can be deterministically drawn by

[TABLE]

where $\sigma$ denotes the sigmoid function and $\tau$ is the temperature that decides how steep the sigmoid function is. If $\tau$ goes to infinite, the above parametrisation is equivalent to drawing the sample from Bernoulli distribution. (Similar relaxation is used in (?) with another re-parametrisation method) By adopting Eq. 16, the derivatives in Eq. 15 can be computed. Combining the Eq. 8, 10 and 14 , one can see that the posterior distribution over the hybrid representation $\boldsymbol{\varphi}$ can be approximated by simply training the super-network in an end-to-end fashion with two additional regularization terms and dropout ratio $p$.

2.3 Sampling and Ranking

Once the variational distribution $q_{\theta}(\boldsymbol{\varphi})$ is obtained, we sample a group of network candidates $S=\{s_{1},s_{2},...,s_{C}\}$ w.r.t. $q_{\theta}(\boldsymbol{\varphi})$, where the $C$ is the number of samples. According to Eq. 13, our sampling process is performed by activating kernels stochastically with the learned $p_{l}^{s}$, which is equivalent to regular dropout operation. Specifically, each candidate is sampled by randomly dropping convolution kernel $w_{l,k}^{s}$ w.r.t. the probability $p_{l}^{s}$ for every $l$, $k$ and $s$ in the super-network model. Then the sampled candidates are evaluated and ranked on a held-out validation dataset. Due to the hybrid network representation, we actually sample architecture-weight pairs, which relieves the mismatch problem. At last, the best-performing one is selected by Eq. 6.

We summarize the complete working flow in Algorithm. 1 and provide proof details in supplementary materials for better understanding. Please note that the proposed PGNAS, though not intentionally, leads to an adaptive dropout that reflects the importance of different parts in the super-network. It thus relieves the dependency on the hyper-parameter sensitive, carefully designed drop-out probability in the previous one-shot methods (?).

3 Experiments

To fully investigate the behavior of the PGNAS, we test our PGNAS on six super-networks. Because we use $u=0$ to facilitate Eq. 15, we construct the super-networks based on architecture priors perceived from manually designed networks. We evaluate the performance of our PGNAS on three databases Cifar-10, Cifar-100 and ImageNet, respectively. For every super-network, we insert a dropout layer after each convolution layer according to Eq. 16 to facilitate the computation of Eq. 15. This modification introduces parameters and FLOPS of negligible overheads. Our PGNAS is trained in an end-to-end way with the Stochastic Gradient Descent (SGD) using a single P40 GPU card for Cifar-10/Cifar-100 and 4 M40 GPU cards for ImageNet. Once a model converges, we sample different convolution kernels w.r.t. the learned dropout ratio to get 1500/5000/1500 candidate architectures for Cifar-10, Cifar-100 and ImageNet, respectively. These 1500 candidates are ranked on a held-out validation dataset and the one with the best performance will be selected as the final search result.

3.1 Cifar-10 and Cifar-100

Super-network and Hyper-parameters. We test our PGNAS on Cifar-10 and Cifar-100 with the super-network, i.e. SupNet-MI, which are based on the manually designed multi-branch ResNet (?). Please refer to the supplementary material for more details of the super-networks and all hyper-parameter settings used in this paper.

Comparison with State-of-the-arts. Table. 1 shows the comparison results on Cifar-10. Here PGNAS-X denotes the performance of our PGNAS on the super-network SupNet-X. From top to bottom, the first group consists of state-of-the-art manually designed architectures on Cifar-10; the following three groups list the related NAS methods utilizing different algorithms, e.g. RL, evolution, and gradient decent; the last group exhibits the performance of PGNAS.

Please note that the search spaces utilized in each work are quite different. For example, (?; ?; ?; ?) employ cell-based search space where each cell contains 4 nodes and there are 5-11 operation candidates between two nodes, (?; ?; ?) utilize DenseNet and PyramidNet (?) as base network, respectively, and (?; ?; ?) apply search algorithm on their self-designed search space. PGNAS-MI incorporates the multi-branch ResNet which only provides up to 4 operation candidates between two nodes (layers) as the initial model. It suggests that our search space is much smaller than the one used in state-of-the-art works such as NAONet, EAS and PathLevel EAS. Still, the proposed PGNAS is capable of finding very advanced architectures in a efficient and effective way, e.g. it finds the architecture at the lowest errors 1.98% on 11.1 GPU days only.

We also enlist the multi-branch ResNet (?) that inspires the design of our super-network in Table 1. Our PGNAS-MI outperform ”shake-shake+cutout” by 0.58%. Regarding the sampling based one-shot method ”One-Shot Top” which achieves a 3.7% classification error by randomly sampling 20000 architectures, our PGNAS attains a much higher performance by sampling only 1500 network architectures due to the posterior distribution guided sampling.

Table. 2 further demonstrate the performance of our PGNAS on a much challenging dataset Cifar-100. Our PGNAS achieves a good trade-off on efficiency and accuracy. It achieves 14.28% error rate with only 11 GPU days, which outperforms the most advanced results NAONet in terms of both model performance and search time.

Please note that results of our PGNAS are achieved during search process without any additional fine-tuning on weights of the searched architectures, while those of other methods are obtained by fine-tuning the searched models. In the following ablation study, we will discuss more on this point.

Ablation Study and Parameter Analysis. We first evaluate the effect of our posterior distribution guided sampling method in Table. 3. In order to demonstrate the generalization of PGNAS, in addition to the SupNet-MI/SupNet-M which are based on the multi-branch ResNet, we also apply PGNAS to the architectures obtained by ENAS (?). We denote them as SupNet-EI/SupNet-E. Please refer to the supplementary materials for more details.

Compared with the baseline ”Random” sampling that is implemented by employing predefined dropout strategy as discussed in (?), PGNAS successfully finds better sub-networks which bring relatively 14% - 23% gain. Evidently, the posterior distribution guided sampling is much more effective, which validates that our approach can learn a meaningful distribution for efficient architecture search. Besides, as can be viewed in the table, there is usually a huge performance gap between the architecture searched with predefined distribution with and without fine-tuning, which reveals the mismatching problems.

Table. 4 discusses the weight prior $l$ in Eq. 16. We find that a good $l$ usually makes the term $\sum_{l,i,s}\frac{(l_{l,k}^{s})^{2}(1-p_{l}^{s})}{2N}$ in Eq. 15 fall into a commonly used weight decay range. So we choose $l$ by grid search. As shown in this table, the weight prior $l$ affects both error rate and model size. The higher the $l$ is, the smaller the size of parameters. We choose the one with the minimal error rate.

Table. 5 shows the impact of temperature value $\tau$ in Eq. 16. It shows that a smaller $\tau$ leads to a lower error, which is consistent with the analysis regarding to Eq. 16. The corresponding fine-tuned result of our PGNAS also provides marginal improvement, which demonstrates the reliability of our PGNAS on sampling of both architecture and weights.

We further evaluate the impact of number of samples in Table. 6. The performance improves along with the increase of number of samples as well as the GPU days. Here we choose sampling 1500 architectures as a trade-off between the complexity and accuracy. Please also note that compared with other sampling-based NAS methods, our scheme achieves 2.17 % error rate by sampling only 50 architectures with the assistance of the estimated a poseteriori distribution. It further reveals the fact that the estimated distribution provides essential information of the distribution of architectures and thus significantly facilitates the sampling process in terms of both efficiency and accuracy.

As discussed before, the correlated L2-like term in the derived objective function Eq. 15 is not a conventional L2 regularization term. As demonstrated in Table. 2, we observe severe performance drop with a constant weighted L2 term.

Visualization. We provide the visualization for the searched architecture of the best-performed PGNAS-MI∗ in supplementary materials. Given the initial super-network a multi-branched ResNet whose block structures are identical, PGNAS can still find diverse structure for each basic block.

3.2 ImageNet

We further evaluate our PGNAS on ImageNet with two super-networks based on ResNet50 (?) and DenseNet121 (?), respectively. Please find detailed settings in the supplementary material. Rather than transferring architectures searched on smaller dataset, the efficiency and flexibility of PGNAS enable us to directly search architectures on ImageNet within few days.

We first provide test results of our PGNAS on ImageNet in Table 9 using a relatively small search space by inflating ResNet50 without limiting the size of the model parameters. Hype-parameters and training process for the three models are identical for fair comparison. It can be observed that PGNAS-R-50 outperforms the ResNet50 by 1.23% with a similar size of parameters. Table. 8 shows the comparison with the state-of-the-art results on ImageNet. Our method can search a very competitive architecture within 26 gpu days. Please note that we do not explicitly control the parameter size of the architectures searched by PGNAS because the goal of PGNAS is to find the architecture with the best accuracy. As can be viewed in Table. 4, larger model does not necessarily generate better performance in our scheme.

Please refer to supplementary materials for more performance evaluation and analysis.

4 Discussions

Weight Sharing. Weight sharing is a popular method adopted by one-shot models to greatly boost the efficiency of NAS. But it is not well understood why sharing weight is effective (?; ?). In PGNAS, as discussed in subsection 2.2, we find that weight sharing can be viewed as a re-parametrization that enables us to estimate the posterior distribution through end-to-end optimization in our scheme.

Network pruning. Our method is a NAS method that conducts channel-level model selection, which is reminiscent of network pruning. We claim that the fundamental goal of NAS and network pruning is quite different. In fact, most of differentiable/one-shot NAS methods such as(?; ?; ?; ?; ?; ?) start from a pre-defined cell/super-net and search for sub-architectures. Channel-level search only enables fine-grained architecture search (e.g., ’channel search space’ in (?)). Besides, it is often that all the channels of an operation are dropped, and PGNAS conducts operation-level selection in such cases. Please refer to Table. 7 for the proportion of channels and operations that are not selected. Actually, in the re-parameterization process of Variational Inference that is used for approximating the posterior distribution, many schemes can be adopted. Dropout, which makes the sampling process resemble pruning, is just one of them. We could also use additive Gaussian noise for re-parameterization and the sampling will not involve pruning any more (section 3.2.2 (?)).

Limitations and Future Works. One limitation of our PGNAS is that it can not explicitly choose the non-parametric operations such as pooling. Another one is that our PGNAS requires prior knowledge on architectures which is hard to achieve. Here we approaches the prior only by manually designed networks. So our future work may be 1) enabling selections on the non-parametric operations (e.g. assigning a 1x1 convolution after each pooling operation as a surrogate to decide whether we need this pooling branch or not.) 2) investigating the robustness of our PGNAS to different prior architectures.

5 Conclusion

In this paper, we view the NAS problem from a Bayesian perspective and propose a new NAS approach, i.e. PGNAS, which converts NAS to a distribution construction problem. It explicitly approximates posterior distribution of network architecture and weights via network training to facilitate an more efficient search process in probability space. It also alleviates the mismatching problem between architecture and shared weights by sampling architecture-weights pair, which provides more reliable ranking results. The proposed PGNAS is efficiently optimized in an end-to-end way, and thus can be easily extended to other large-scale tasks.

Acknowledgement

This work was supported by the National Key R&D Program of China under Grant 2017YFB1300201, the National Natural Science Foundation of China (NSFC) under Grants 61622211 and 61620106009 as well as the Fundamental Research Funds for the Central Universities under Grant WK2100100030.