Neural Inhibition Improves Dynamic Routing and Mixture of Experts

TL;DR

This paper introduces neural inhibition as a novel mechanism to enhance dynamic routing and mixture-of-experts models, leading to improved performance in neural network architectures.

Contribution

It proposes a new neural inhibition technique for dynamic routing models, demonstrating its effectiveness in boosting task performance and encouraging further research in this area.

Findings

Neural inhibition significantly improves model performance.

Inhibition enables better selection of neural pathways.

Experimental results validate the effectiveness of the proposed method.

Abstract

To be effective, efficient, and diverse, deep learning models need to dynamically choose its architecture based on signals from a population of neurons. We hypothesize dynamic routing models can be improved with neural inhibition in those neural populations. This means signals commonly shared among the various modes of data statistics can be inhibited so that the routing model can choose a specialized expert path for each data sample. Only through inhibition is the routing mechanism able to effectively select neural pathways. We believe this is an under-studied and under-verified implementation methodology for Mixture-of-Experts, dynamic routing, and transformer language models. We provide experimental evidence that the neural inhibition algorithm significantly boosts the performance of general tasks and motivates more effort to be invested in this research direction.