Towards automated neural design: An open source, distributed neural architecture research framework

TL;DR

NORD is an open source, distributed framework built on PyTorch, MPI, and Horovod that simplifies neural architecture research by enabling easy implementation, comparison, and optimization of deep learning models across distributed systems.

Contribution

It introduces a flexible, distributed research framework that facilitates neural architecture design, optimization, and comparison with minimal overhead for researchers.

Findings

Framework supports various optimization algorithms.

Enables fair comparison of architectures.

Demonstrates initial experimental results.

Abstract

NORD (Neural Operations Research & Development) is an open source distributed deep learning architectural research framework, based on PyTorch, MPI and Horovod. It aims to make research of deep architectures easier for experts of different domains, in order to accelerate the process of finding better architectures, as well as study the best architectures generated for different datasets. Although currently under heavy development, the framework aims to allow the easy implementation of different design and optimization method families (optimization algorithms, meta-heuristics, reinforcement learning etc.) as well as the fair comparison between them. Furthermore, due to the computational resources required in order to optimize and evaluate network architectures, it leverage the use of distributed computing, while aiming to minimize the researcher's overhead required to implement it. Moreover, it strives to make the creation of architectures more intuitive, by implementing network descriptors, allowing to separately define the architecture's nodes and connections. In this paper, we present the framework's current state of development, while presenting its basic concepts, providing simple examples as well as their experimental results.