Leaky ReLUs That Differ in Forward and Backward Pass Facilitate Activation Maximization in Deep Neural Networks

TL;DR

This paper introduces a novel activation maximization method using Leaky ReLUs with different forward and backward slopes, improving interpretability and training performance of neural networks.

Contribution

It proposes a new approach with Leaky ReLUs in the backward pass and introduces ProxyGrad, a secondary network technique for better gradient estimation and network training.

Findings

AM fails with standard ReLUs, but improves with Leaky ReLUs in the backward pass.

ProxyGrad outperforms traditional training methods on several benchmarks.

Using different slopes in Leaky ReLUs enhances interpretability and training efficiency.

Abstract

Activation maximization (AM) strives to generate optimal input stimuli, revealing features that trigger high responses in trained deep neural networks. AM is an important method of explainable AI. We demonstrate that AM fails to produce optimal input stimuli for simple functions containing ReLUs or Leaky ReLUs, casting doubt on the practical usefulness of AM and the visual interpretation of the generated images. This paper proposes a solution based on using Leaky ReLUs with a high negative slope in the backward pass while keeping the original, usually zero, slope in the forward pass. The approach significantly increases the maxima found by AM. The resulting ProxyGrad algorithm implements a novel optimization technique for neural networks that employs a secondary network as a proxy for gradient computation. This proxy network is designed to have a simpler loss landscape with fewer local maxima than the original network. Our chosen proxy network is an identical copy of the original network, including its weights, with distinct negative slopes in the Leaky ReLUs. Moreover, we show that ProxyGrad can be used to train the weights of Convolutional Neural Networks for classification such that, on some of the tested benchmarks, they outperform traditional networks.