Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning

TL;DR

This paper demonstrates that modular recurrent neural networks, trained via an iterative growth curriculum, outperform non-modular networks in training efficiency, generalization, and robustness, highlighting the benefits of structural modularity.

Contribution

It introduces a modular growth curriculum for RNNs, showing improved performance and robustness, and analyzes how modular connectivity influences computational capabilities.

Findings

Modular networks outperform non-modular RNNs in multiple metrics.

Inductive bias from modular topology enables effective training with fixed intra-module connections.

Gradual modular growth aids in learning complex tasks and building scalable networks.

Abstract

Structural modularity is a pervasive feature of biological neural networks, which have been linked to several functional and computational advantages. Yet, the use of modular architectures in artificial neural networks has been relatively limited despite early successes. Here, we explore the performance and functional dynamics of a modular network trained on a memory task via an iterative growth curriculum. We find that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics, including training time, generalizability, and robustness to some perturbations. We further examine how different aspects of a modular network's connectivity contribute to its computational capability. We then demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed and only the connections between modules are trained. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales, and help build more scalable and compressible artificial networks.