Variational Adaptive Noise and Dropout towards Stable Recurrent Neural Networks

TL;DR

This paper introduces VAND, a novel theory combining adaptive noise and dropout for stable RNN training, enabling better imitation of complex behaviors in robotics.

Contribution

It presents a unified variational inference framework for adaptive noise and dropout, enhancing RNN stability and performance.

Findings

VAND achieves stable RNN training.

VAND enables imitation of complex behaviors.

Improved RNN robustness demonstrated.

Abstract

This paper proposes a novel stable learning theory for recurrent neural networks (RNNs), so-called variational adaptive noise and dropout (VAND). As stabilizing factors for RNNs, noise and dropout on the internal state of RNNs have been separately confirmed in previous studies. We reinterpret the optimization problem of RNNs as variational inference, showing that noise and dropout can be derived simultaneously by transforming the explicit regularization term arising in the optimization problem into implicit regularization. Their scale and ratio can also be adjusted appropriately to optimize the main objective of RNNs, respectively. In an imitation learning scenario with a mobile manipulator, only VAND is able to imitate sequential and periodic behaviors as instructed. https://youtu.be/UOho3Xr6A2w