Sub-Architecture Ensemble Pruning in Neural Architecture Search

TL;DR

This paper introduces SAEP, a pruning method for neural architecture search ensembles that enhances diversity and reduces the number of sub-architectures while maintaining performance.

Contribution

It proposes a novel pruning approach in NAS ensembles to improve diversity and efficiency, addressing redundancy issues in ensemble selection.

Findings

Significantly reduces sub-architecture count without performance loss

Enhances ensemble diversity in NAS

Demonstrates effectiveness through extensive experiments

Abstract

Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and remarkable capability to reduce the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computations that might not be affordable for researchers and practitioners. While recent attempts have employed ensemble learning methods to mitigate the enormous computational cost, however, they neglect a key property of ensemble methods, namely diversity, which leads to collecting more similar sub-architectures with potential redundancy in the final design. To tackle this problem, we propose a pruning method for NAS ensembles called "Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)." It targets to leverage diversity and to achieve sub-ensemble architectures at a smaller size with comparable performance to ensemble architectures that are not pruned. Three possible solutions are proposed to decide which sub-architectures to prune during the searching process. Experimental results exhibit the effectiveness of the proposed method by largely reducing the number of sub-architectures without degrading the performance.