Exploring epoch-dependent stochastic residual networks

TL;DR

This paper investigates epoch-dependent stochastic residual networks, where the probability of bypassing layers changes over training epochs, aiming to improve training dynamics and model performance.

Contribution

It introduces the concept of epoch-dependent distributions in stochastic residual networks, a novel approach that adjusts layer activation probabilities over training.

Findings

Preliminary results show mixed outcomes.

Epoch-dependent strategies have potential for further research.

The approach offers a new dimension for controlling stochastic residual networks.

Abstract

The recently proposed stochastic residual networks selectively activate or bypass the layers during training, based on independent stochastic choices, each of which following a probability distribution that is fixed in advance. In this paper we present a first exploration on the use of an epoch-dependent distribution, starting with a higher probability of bypassing deeper layers and then activating them more frequently as training progresses. Preliminary results are mixed, yet they show some potential of adding an epoch-dependent management of distributions, worth of further investigation.