Multi-Bias Non-linear Activation in Deep Neural Networks

TL;DR

This paper introduces a multi-bias activation layer that enhances feature representation in deep neural networks by multi-thresholding response magnitudes, leading to improved performance with low computational cost.

Contribution

It proposes a novel multi-bias non-linear activation (MBA) layer that decouples responses into multiple maps, enabling richer feature representations at a low computational cost.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Provides flexible response selection for different visual patterns.

Enhances feature diversity without increasing model complexity.

Abstract

As a widely used non-linear activation, Rectified Linear Unit (ReLU) separates noise and signal in a feature map by learning a threshold or bias. However, we argue that the classification of noise and signal not only depends on the magnitude of responses, but also the context of how the feature responses would be used to detect more abstract patterns in higher layers. In order to output multiple response maps with magnitude in different ranges for a particular visual pattern, existing networks employing ReLU and its variants have to learn a large number of redundant filters. In this paper, we propose a multi-bias non-linear activation (MBA) layer to explore the information hidden in the magnitudes of responses. It is placed after the convolution layer to decouple the responses to a convolution kernel into multiple maps by multi-thresholding magnitudes, thus generating more patterns in the feature space at a low computational cost. It provides great flexibility of selecting responses to different visual patterns in different magnitude ranges to form rich representations in higher layers. Such a simple and yet effective scheme achieves the state-of-the-art performance on several benchmarks.