Emergence of Functionally Differentiated Structures via Mutual Information Minimization in Recurrent Neural Networks

TL;DR

This paper introduces a method for inducing functional differentiation in recurrent neural networks by minimizing mutual information, leading to early emergence of functional modules before structural changes, offering insights into brain-like specialization.

Contribution

The study presents a novel mutual information minimization approach to promote functional differentiation in RNNs, linking information theory with neural modularity development.

Findings

Functional differentiation emerges earlier than structural modularity.

Mutual information minimization improves task performance.

Functional modules are identified through correlation patterns.

Abstract

Functional differentiation in the brain emerges as distinct regions specialize and is key to understanding brain function as a complex system. Previous research has modeled this process using artificial neural networks with specific constraints. Here, we propose a novel approach that induces functional differentiation in recurrent neural networks by minimizing mutual information between neural subgroups via mutual information neural estimation. We apply our method to a 2-bit working memory task and a chaotic signal separation task involving Lorenz and R\"ossler time series. Analysis of network performance, correlation patterns, and weight matrices reveals that mutual information minimization yields high task performance alongside clear functional modularity and moderate structural modularity. Importantly, our results show that functional differentiation, which is measured through correlation structures, emerges earlier than structural modularity defined by synaptic weights. This suggests that functional specialization precedes and probably drives structural reorganization within developing neural networks. Our findings provide new insights into how information-theoretic principles may govern the emergence of specialized functions and modular structures during artificial and biological brain development.