LogAvgExp Provides a Principled and Performant Global Pooling Operator

TL;DR

This paper introduces LogAvgExp, a theoretically justified and flexible global pooling operator for neural networks that smoothly transitions between max and mean pooling, improving performance across vision tasks.

Contribution

The paper proposes LogAvgExp as a principled pooling operator with a learnable temperature, unifying max and mean pooling within a single framework.

Findings

LogAvgExp acts as a natural OR operator for logits.

Introducing a temperature parameter allows smooth transition between max and mean pooling.

Experimental results show improved performance in vision neural networks.

Abstract

We seek to improve the pooling operation in neural networks, by applying a more theoretically justified operator. We demonstrate that LogSumExp provides a natural OR operator for logits. When one corrects for the number of elements inside the pooling operator, this becomes $\text{LogAvgExp} := \log(\text{mean}(\exp(x)))$. By introducing a single temperature parameter, LogAvgExp smoothly transitions from the max of its operands to the mean (found at the limiting cases $t \to 0^+$ and $t \to +\infty$). We experimentally tested LogAvgExp, both with and without a learnable temperature parameter, in a variety of deep neural network architectures for computer vision.