Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learning

TL;DR

This paper investigates the complex relationship between modularity and performance in neural networks for reinforcement learning, revealing that optimal modularity is entangled with many network and environment features, making direct optimization challenging.

Contribution

It demonstrates that optimal modularity in neural networks is complexly intertwined with other features, and that automatic neuroevolutionary methods reveal intricate relationships affecting performance.

Findings

Optimal network modularity is entangled with other features.

Direct optimization of modularity may not improve performance.

Complex relationships influence the effectiveness of modularity in neural networks.

Abstract

Modularity has been widely studied as a mechanism to improve the capabilities of neural networks through various techniques such as hand-crafted modular architectures and automatic approaches. While these methods have sometimes shown improvements towards generalisation ability, robustness, and efficiency, the mechanisms that enable modularity to give performance advantages are unclear. In this paper, we investigate this issue and find that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment. Therefore, direct optimisation or arbitrary designation of a suitable amount of modularity in neural networks may not be beneficial. We used a classic neuroevolutionary algorithm which enables rich, automatic optimisation and exploration of neural network architectures and weights with varying levels of modularity. The structural modularity and performance of networks generated by the NeuroEvolution of Augmenting Topologies algorithm was assessed on three reinforcement learning tasks, with and without an additional modularity objective. The results of the quality-diversity optimisation algorithm, MAP-Elites, suggest intricate conditional relationships between modularity, performance, and other predefined network features.