Axiomatization of Gradient Smoothing in Neural Networks

TL;DR

This paper introduces a theoretical framework for gradient smoothing in neural networks, based on function mollification and Monte Carlo integration, providing a foundation for understanding and designing smoothing methods.

Contribution

It axiomatizes gradient smoothing in neural networks and offers a systematic approach to create new smoothing techniques based on the framework.

Findings

Framework reveals the rationale behind existing smoothing methods.

New smoothing methods designed from the framework show promising experimental results.

Research potential demonstrated through experimental validation.

Abstract

Gradients play a pivotal role in neural networks explanation. The inherent high dimensionality and structural complexity of neural networks result in the original gradients containing a significant amount of noise. While several approaches were proposed to reduce noise with smoothing, there is little discussion of the rationale behind smoothing gradients in neural networks. In this work, we proposed a gradient smooth theoretical framework for neural networks based on the function mollification and Monte Carlo integration. The framework intrinsically axiomatized gradient smoothing and reveals the rationale of existing methods. Furthermore, we provided an approach to design new smooth methods derived from the framework. By experimental measurement of several newly designed smooth methods, we demonstrated the research potential of our framework.