# Variance Networks: When Expectation Does Not Meet Your Expectations

## TL;DR

This paper introduces variance layers in neural networks, where weights are zero-mean with learnable variance, demonstrating their effectiveness in exploration, robustness, and as a Bayesian approximation, challenging traditional mean-based approaches.

## Contribution

The paper proposes variance layers with zero-mean weights, showing their advantages in exploration, adversarial defense, and Bayesian neural network training.

## Key findings

- Variance layers perform well in reinforcement learning exploration.
- Zero-mean Bayesian neural networks converge to better training objectives.
- Variance layers offer robustness against adversarial attacks.

## Abstract

Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than conventional parameterizations where the mean is being learned.