Generative flow induced neural architecture search: Towards discovering optimal architecture in wavelet neural operator

TL;DR

This paper introduces a generative flow-based neural architecture search method that efficiently discovers optimal wavelet neural operator architectures by learning stochastic policies, outperforming traditional grid search and reinforcement learning approaches.

Contribution

The paper presents a novel flow-induced neural architecture search algorithm that generates hyperparameters sequentially, reducing exploration time and improving wavelet neural operator performance.

Findings

The method outperforms grid search in architecture discovery.

It improves wavelet neural operator accuracy across fluid mechanics problems.

The approach reduces exploration time compared to reinforcement learning.

Abstract

We propose a generative flow-induced neural architecture search algorithm. The proposed approach devices simple feed-forward neural networks to learn stochastic policies to generate sequences of architecture hyperparameters such that the generated states are in proportion with the reward from the terminal state. We demonstrate the efficacy of the proposed search algorithm on the wavelet neural operator (WNO), where we learn a policy to generate a sequence of hyperparameters like wavelet basis and activation operators for wavelet integral blocks. While the trajectory of the generated wavelet basis and activation sequence is cast as flow, the policy is learned by minimizing the flow violation between each state in the trajectory and maximizing the reward from the terminal state. In the terminal state, we train WNO simultaneously to guide the search. We propose to use the exponent of the negative of the WNO loss on the validation dataset as the reward function. While the grid search-based neural architecture generation algorithms foresee every combination, the proposed framework generates the most probable sequence based on the positive reward from the terminal state, thereby reducing exploration time. Compared to reinforcement learning schemes, where complete episodic training is required to get the reward, the proposed algorithm generates the hyperparameter trajectory sequentially. Through four fluid mechanics-oriented problems, we illustrate that the learned policies can sample the best-performing architecture of the neural operator, thereby improving the performance of the vanilla wavelet neural operator.