Modularity as a Means for Complexity Management in Neural Networks Learning

TL;DR

This paper introduces a modular neural network design that decomposes complex models into control and functional modules, improving training efficiency and stability, demonstrated on a list sorting task.

Contribution

It presents the first modular approach to neural network design, addressing complexity management and optimization challenges in large, intricate models.

Findings

Modular NNs train faster than monolithic counterparts.

Modular NNs show increased training stability.

Modular design enhances maintainability.

Abstract

Training a Neural Network (NN) with lots of parameters or intricate architectures creates undesired phenomena that complicate the optimization process. To address this issue we propose a first modular approach to NN design, wherein the NN is decomposed into a control module and several functional modules, implementing primitive operations. We illustrate the modular concept by comparing performances between a monolithic and a modular NN on a list sorting problem and show the benefits in terms of training speed, training stability and maintainability. We also discuss some questions that arise in modular NNs.