A Deeper Look at Power Normalizations

TL;DR

This paper introduces a novel power normalization layer for deep learning that improves feature pooling in CNNs, leading to state-of-the-art results in various recognition tasks.

Contribution

It proposes a new kernel-based power normalization layer for CNNs, with a probabilistic interpretation and well-behaved gradients, enhancing feature pooling capabilities.

Findings

Achieves state-of-the-art performance on four benchmarks.

Demonstrates effectiveness of the power normalization layer in CNNs.

Provides a probabilistic understanding of power normalization functions.

Abstract

Power Normalizations (PN) are very useful non-linear operators in the context of Bag-of-Words data representations as they tackle problems such as feature imbalance. In this paper, we reconsider these operators in the deep learning setup by introducing a novel layer that implements PN for non-linear pooling of feature maps. Specifically, by using a kernel formulation, our layer combines the feature vectors and their respective spatial locations in the feature maps produced by the last convolutional layer of CNN. Linearization of such a kernel results in a positive definite matrix capturing the second-order statistics of the feature vectors, to which PN operators are applied. We study two types of PN functions, namely (i) MaxExp and (ii) Gamma, addressing their role and meaning in the context of nonlinear pooling. We also provide a probabilistic interpretation of these operators and derive their surrogates with well-behaved gradients for end-to-end CNN learning. We apply our theory to practice by implementing the PN layer on a ResNet-50 model and showcase experiments on four benchmarks for fine-grained recognition, scene recognition, and material classification. Our results demonstrate state-of-the-art performance across all these tasks.