Hadamard Representations: Augmenting Hyperbolic Tangents in RL

TL;DR

This paper introduces Hadamard representations to enhance hyperbolic tangent activations in reinforcement learning, addressing dying neuron issues and improving learning efficiency and representation capacity.

Contribution

It proposes a novel Hadamard representation method that mitigates dying neurons in continuous activations like tanh, boosting RL performance.

Findings

Faster learning in Atari benchmarks

Reduced dead neurons in neural networks

Increased effective rank of activations

Abstract

Activation functions are one of the key components of a deep neural network. The most commonly used activation functions can be classed into the category of continuously differentiable (e.g. tanh) and piece-wise linear functions (e.g. ReLU), both having their own strengths and drawbacks with respect to downstream performance and representation capacity through learning. In reinforcement learning, the performance of continuously differentiable activations often falls short as compared to piece-wise linear functions. We show that the dying neuron problem in RL is not exclusive to ReLUs and actually leads to additional problems in the case of continuously differentiable activations such as tanh. To alleviate the dying neuron problem with these activations, we propose a Hadamard representation that unlocks the advantages of continuously differentiable activations. Using DQN, PPO and PQN in the Atari domain, we show faster learning, a reduction in dead neurons and increased effective rank.