Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architectures

TL;DR

This paper introduces a morphism-based, transformable backbone architecture for differentiable neural architecture search, enabling dynamic growth during training and improving efficiency and performance in recurrent neural networks.

Contribution

It proposes a novel growing mechanism based on network morphism for differentiable NAS, allowing adaptive architecture growth during training.

Findings

Effective in multi-variate time series forecasting.

Outperforms baseline architectures like LSTM.

Enhances efficiency of differentiable architecture search.

Abstract

This study aims at making the architecture search process more adaptive for one-shot or online training. It is extended from the existing study on differentiable neural architecture search, and we made the backbone architecture transformable rather than fixed during the training process. As is known, differentiable neural architecture search (DARTS) requires a pre-defined over-parameterized backbone architecture, while its size is to be determined manually. Also, in DARTS backbone, Hadamard product of two elements is not introduced, which exists in both LSTM and GRU cells for recurrent nets. This study introduces a growing mechanism for differentiable neural architecture search based on network morphism. It enables growing of the cell structures from small size towards large size ones with one-shot training. Two modes can be applied in integrating the growing and original pruning process. We also implement a recently proposed two-input backbone architecture for recurrent neural networks. Initial experimental results indicate that our approach and the two-input backbone structure can be quite effective compared with other baseline architectures including LSTM, in a variety of learning tasks including multi-variate time series forecasting and language modeling. On the other hand, we find that dynamic network transformation is promising in improving the efficiency of differentiable architecture search.