Uncertainty Propagation in Convolutional Neural Networks: Technical Report

TL;DR

This technical report investigates how uncertainty, modeled as variances of normal variables, propagates through various CNN layers including linear, non-linear, and loss functions, providing analytical approximations.

Contribution

It offers a systematic analysis of uncertainty propagation in CNN components, including new approximations for moments of non-linear functions like sigmoid and loss functions.

Findings

Derived formulas for variance propagation in convolutional and pooling layers.

Provided approximations for moments of sigmoid and cross-entropy loss under uncertainty.

Enhanced understanding of uncertainty behavior in CNNs for probabilistic modeling.

Abstract

In this technical report we study the problem of propagation of uncertainty (in terms of variances of given uni-variate normal random variables) through typical building blocks of a Convolutional Neural Network (CNN). These include layers that perform linear operations, such as 2D convolutions, fully-connected, and average pooling layers, as well as layers that act non-linearly on their input, such as the Rectified Linear Unit (ReLU). Finally, we discuss the sigmoid function, for which we give approximations of its first- and second-order moments, as well as the binary cross-entropy loss function, for which we approximate its expected value under normal random inputs.