Convolutional Normalization

TL;DR

This paper introduces a novel normalization technique for deep neural networks that employs weighted sums via depth-wise convolutional neural networks to better approximate complex data statistics, aiming to improve training efficiency.

Contribution

It proposes a new normalization method using learnable weighted sums with depth-wise CNNs, addressing limitations of traditional Monte Carlo-based normalization techniques.

Findings

Enhanced approximation of complex data statistics.

Potential improvements in training speed and efficiency.

Learned coefficients adapt to data distribution.

Abstract

As the deep neural networks are being applied to complex tasks, the size of the networks and architecture increases and their topology becomes more complicated too. At the same time, training becomes slow and at some instances inefficient. This motivated the introduction of various normalization techniques such as Batch Normalization and Layer Normalization. The aforementioned normalization methods use arithmetic operations to compute an approximation statistics (mainly the first and second moments) of the layer's data and use it to normalize it. The aforementioned methods use plain Monte Carlo method to approximate the statistics and such method fails when approximating the statistics whose distribution is complex. Here, we propose an approach that uses weighted sum, implemented using depth-wise convolutional neural networks, to not only approximate the statistics, but to learn the coefficients of the sum.