Density-embedding layers: a general framework for adaptive receptive fields

TL;DR

This paper introduces density-embedded layers, a new theoretical framework that generalizes neuron transformations to enable adaptive and trainable receptive fields, improving neural network flexibility for visual tasks.

Contribution

The paper proposes a novel density-embedded layer framework that generalizes neuron transformations, allowing efficient training and fine-tuning of receptive fields in neural networks.

Findings

Framework captures and generalizes recent adaptive methods

Efficient training via automatic differentiation

Validated on MNIST dataset

Abstract

The effectiveness and performance of artificial neural networks, particularly for visual tasks, depends in crucial ways on the receptive field of neurons. The receptive field itself depends on the interplay between several architectural aspects, including sparsity, pooling, and activation functions. In recent literature there are several ad hoc proposals trying to make receptive fields more flexible and adaptive to data. For instance, different parameterizations of convolutional and pooling layers have been proposed to increase their adaptivity. In this paper, we propose the novel theoretical framework of density-embedded layers, generalizing the transformation represented by a neuron. Specifically, the affine transformation applied on the input is replaced by a scalar product of the input, suitably represented as a piecewise constant function, with a density function associated with the neuron. This density is shown to describe directly the receptive field of the neuron. Crucially, by suitably representing such a density as a linear combination of a parametric family of functions, we can efficiently train the densities by means of any automatic differentiation system, making it adaptable to the problem at hand, and computationally efficient to evaluate. This framework captures and generalizes recent methods, allowing a fine tuning of the receptive field. In the paper, we define some novel layers and we experimentally validate them on the classic MNIST dataset.