Brain-Like Processing Pathways Form in Models With Heterogeneous Experts

TL;DR

This paper introduces a new model demonstrating how biologically inspired biases lead to the formation of task-specific pathways in neural networks, mirroring brain organization during learning.

Contribution

It proposes a framework with inductive biases that promote pathway formation in heterogeneous models, aligning artificial pathways with brain function.

Findings

Artificial pathways match brain's cortical and subcortical interactions

Pathway formation depends on routing costs and task difficulty

Model explains how brain regions organize into pathways

Abstract

The brain is made up of a vast set of heterogeneous regions that dynamically organize into pathways as a function of task demands. Examples of such pathways can be found in the interactions between cortical and subcortical networks during learning, or in sub-networks specializing for task characteristics such as difficulty or modality. Despite the large role these pathways play in cognition, the mechanisms through which brain regions organize into pathways remain unclear. In this work, we use an extension of the Heterogeneous Mixture-of-Experts architecture to show that heterogeneous regions do not form processing pathways by themselves, implying that the brain likely implements specific constraints which result in the reliable formation of pathways. We identify three biologically relevant inductive biases that encourage pathway formation: a routing cost imposed on the use of more complex regions, a scaling factor that reduces this cost when task performance is low, and randomized expert dropout. When comparing our resulting \textit{Mixture-of-Pathways} model with the brain, we observe that the artificial pathways in our model match how the brain uses cortical and subcortical systems to learn and solve tasks of varying difficulty. In summary, we introduce a novel framework for investigating how the brain forms task-specific pathways through inductive biases, and the effects these biases have on the behavior of Mixture-of-Experts models.