RLFlow: Optimising Neural Network Subgraph Transformation with World Models

TL;DR

RLFlow employs a model-based reinforcement learning approach with World Models to optimize neural network subgraph transformations, reducing runtime without relying on handcrafted heuristics, and achieves competitive or superior performance.

Contribution

This paper introduces RLFlow, a novel model-based RL method using World Models for neural network optimization, enhancing sample efficiency and performance over existing approaches.

Findings

RLFlow matches state-of-the-art on convolutional networks.

RLFlow outperforms transformer architectures by up to 5%.

Model-based RL with World Models effectively optimizes neural network subgraphs.

Abstract

Training deep learning models takes an extremely long execution time and consumes large amounts of computing resources. At the same time, recent research proposed systems and compilers that are expected to decrease deep learning models runtime. An effective optimisation methodology in data processing is desirable, and the reduction of compute requirements of deep learning models is the focus of extensive research. In this paper, we address the neural network sub-graph transformation by exploring reinforcement learning (RL) agents to achieve performance improvement. Our proposed approach RLFlow can learn to perform neural network subgraph transformations, without the need for expertly designed heuristics to achieve a high level of performance. Recent work has aimed at applying RL to computer systems with some success, especially using model-free RL techniques. Model-based reinforcement learning methods have seen an increased focus in research as they can be used to learn the transition dynamics of the environment; this can be leveraged to train an agent using a hallucinogenic environment such as World Model (WM), thereby increasing sample efficiency compared to model-free approaches. WM uses variational auto-encoders and it builds a model of the system and allows exploring the model in an inexpensive way. In RLFlow, we propose a design for a model-based agent with WM which learns to optimise the architecture of neural networks by performing a sequence of sub-graph transformations to reduce model runtime. We show that our approach can match the state-of-the-art performance on common convolutional networks and outperforms by up to 5% those based on transformer-style architectures