GreedyNASv2: Greedier Search with a Greedy Path Filter

TL;DR

GreedyNASv2 introduces an explicit path filter trained via PU learning to efficiently identify promising paths in NAS, significantly improving search efficiency and resulting in high-accuracy models on ImageNet.

Contribution

It proposes a novel path filtering method using PU learning and path embeddings to shrink the search space and enhance NAS efficiency and accuracy.

Findings

Achieves 81.1% Top-1 accuracy on ImageNet with GreedyNASv2-L.

Outperforms ResNet-50 baseline significantly.

Demonstrates effective path filtering and embedding strategies.

Abstract

Training a good supernet in one-shot NAS methods is difficult since the search space is usually considerably huge (e.g., $13^{21}$). In order to enhance the supernet's evaluation ability, one greedy strategy is to sample good paths, and let the supernet lean towards the good ones and ease its evaluation burden as a result. However, in practice the search can be still quite inefficient since the identification of good paths is not accurate enough and sampled paths still scatter around the whole search space. In this paper, we leverage an explicit path filter to capture the characteristics of paths and directly filter those weak ones, so that the search can be thus implemented on the shrunk space more greedily and efficiently. Concretely, based on the fact that good paths are much less than the weak ones in the space, we argue that the label of "weak paths" will be more confident and reliable than that of "good paths" in multi-path sampling. In this way, we thus cast the training of path filter in the positive and unlabeled (PU) learning paradigm, and also encourage a \textit{path embedding} as better path/operation representation to enhance the identification capacity of the learned filter. By dint of this embedding, we can further shrink the search space by aggregating similar operations with similar embeddings, and the search can be more efficient and accurate. Extensive experiments validate the effectiveness of the proposed method GreedyNASv2. For example, our obtained GreedyNASv2-L achieves $81.1\%$ Top-1 accuracy on ImageNet dataset, significantly outperforming the ResNet-50 strong baselines.