GitGraph - Architecture Search Space Creation through Frequent Computational Subgraph Mining

TL;DR

This paper introduces GitGraph, a method to create neural architecture search spaces by mining common subgraphs from existing neural networks, enabling more flexible and efficient architecture design.

Contribution

It proposes a novel approach to generate problem-specific neural search spaces by mining and utilizing common subgraphs from a large corpus of architectures.

Findings

GitGraph corpus of architectures and descriptions published

Problem-specific search spaces created via textual search over GitGraph

Unique common subgraphs identified as modules for architecture design

Abstract

The dramatic success of deep neural networks across multiple application areas often relies on experts painstakingly designing a network architecture specific to each task. To simplify this process and make it more accessible, an emerging research effort seeks to automate the design of neural network architectures, using e.g. evolutionary algorithms or reinforcement learning or simple search in a constrained space of neural modules. Considering the typical size of the search space (e.g. $10^{10}$ candidates for a $10$-layer network) and the cost of evaluating a single candidate, current architecture search methods are very restricted. They either rely on static pre-built modules to be recombined for the task at hand, or they define a static hand-crafted framework within which they can generate new architectures from the simplest possible operations. In this paper, we relax these restrictions, by capitalizing on the collective wisdom contained in the plethora of neural networks published in online code repositories. Concretely, we (a) extract and publish GitGraph, a corpus of neural architectures and their descriptions; (b) we create problem-specific neural architecture search spaces, implemented as a textual search mechanism over GitGraph; (c) we propose a method of identifying unique common subgraphs within the architectures solving each problem (e.g., image processing, reinforcement learning), that can then serve as modules in the newly created problem specific neural search space.