Conditional Activation for Diverse Neurons in Heterogeneous Networks

TL;DR

This paper introduces conditional activation, a dynamic neuron behavior modeling scheme, applied to heterogeneous neural networks, leading to improved learning speed, performance, and reduced memory usage compared to traditional homogeneous networks.

Contribution

The paper presents a novel conditional activation method that enables neurons to dynamically modify their activation functions, enhancing heterogeneity and performance in neural networks.

Findings

Improved learning speed and performance in heterogeneous MLPs

Reduced memory requirements for network parameters

Effective modeling of special neurons in sensory systems

Abstract

In this paper, we propose a new scheme for modelling the diverse behavior of neurons. We introduce the conditional activation, in which a neurons activation function is dynamically modified by a control signal. We apply this method to recreate behavior of special neurons existing in the human auditory and visual system. A heterogeneous multilayered perceptron (MLP) incorporating the developed models demonstrates simultaneous improvement in learning speed and performance across a various number of hidden units and layers, compared to a homogeneous network composed of the conventional neuron model. For similar performance, the proposed model lowers the memory for storing network parameters significantly.