IC Neuron: An Efficient Unit to Construct Neural Networks

TL;DR

This paper introduces the IC neuron, a novel neural unit inspired by physics, which enhances the representation capacity of neural networks and improves performance across various architectures.

Contribution

The paper proposes the IC neuron model that divides input space into subspaces for better representation, integrating it into different network types to outperform traditional models.

Findings

IC networks outperform traditional networks in experiments

IC neuron enhances non-linear representation ability

IC neuron emphasizes useful input features

Abstract

As a popular machine learning method, neural networks can be used to solve many complex tasks. Their strong generalization ability comes from the representation ability of the basic neuron model. The most popular neuron is the MP neuron, which uses a linear transformation and a non-linear activation function to process the input successively. Inspired by the elastic collision model in physics, we propose a new neuron model that can represent more complex distributions. We term it Inter-layer collision (IC) neuron. The IC neuron divides the input space into multiple subspaces used to represent different linear transformations. This operation enhanced non-linear representation ability and emphasizes some useful input features for the given task. We build the IC networks by integrating the IC neurons into the fully-connected (FC), convolutional, and recurrent structures. The IC networks outperform the traditional networks in a wide range of experiments. We believe that the IC neuron can be a basic unit to build network structures.