Optimizing Recurrent Neural Networks Architectures under Time Constraints

TL;DR

This paper introduces algorithms to optimize RNN architectures under time constraints by transforming the problem into a submodular optimization, resulting in more accurate or faster models compared to existing methods.

Contribution

It presents a novel approach converting RNN architecture optimization into a submodular subset selection problem with bounds, improving efficiency and effectiveness.

Findings

Algorithms outperform manual tuning and random search.

Optimized models are more accurate or faster.

Comparison of RNN architectures using the proposed methods.

Abstract

Recurrent neural network (RNN)'s architecture is a key factor influencing its performance. We propose algorithms to optimize hidden sizes under running time constraint. We convert the discrete optimization into a subset selection problem. By novel transformations, the objective function becomes submodular and constraint becomes supermodular. A greedy algorithm with bounds is suggested to solve the transformed problem. And we show how transformations influence the bounds. To speed up optimization, surrogate functions are proposed which balance exploration and exploitation. Experiments show that our algorithms can find more accurate models or faster models than manually tuned state-of-the-art and random search. We also compare popular RNN architectures using our algorithms.