Receptive Field Size Optimization with Continuous Time Pooling

TL;DR

This paper introduces a novel continuous time pooling method for CNNs, replacing traditional max pooling with a differential equation approach that allows for gradient-based optimization of receptive field size, aiming to improve robustness and invariance.

Contribution

It presents a continuous time pooling technique that models receptive fields via differential equations, enabling gradient-based optimization of pooling parameters.

Findings

Continuous pooling improves network robustness.

Gradient optimization of pooling parameters is feasible.

Method is compatible with GPU implementations.

Abstract

The pooling operation is a cornerstone element of convolutional neural networks. These elements generate receptive fields for neurons, in which local perturbations should have minimal effect on the output activations, increasing robustness and invariance of the network. In this paper we will present an altered version of the most commonly applied method, maximum pooling, where pooling in theory is substituted by a continuous time differential equation, which generates a location sensitive pooling operation, more similar to biological receptive fields. We will present how this continuous method can be approximated numerically using discrete operations which fit ideally on a GPU. In our approach the kernel size is substituted by diffusion strength which is a continuous valued parameter, this way it can be optimized by gradient descent algorithms. We will evaluate the effect of continuous pooling on accuracy and computational need using commonly applied network architectures and datasets.