AdaptGrad: Adaptive Sampling to Reduce Noise

TL;DR

AdaptGrad introduces an adaptive sampling method for gradient smoothing that effectively reduces noise in gradient explanations, improving interpretability without manual hyper-parameter tuning.

Contribution

The paper reinterprets SmoothGrad as a convolution process and proposes AdaptGrad, an adaptive method that automatically adjusts noise levels for better gradient noise reduction.

Findings

AdaptGrad significantly reduces noise compared to baseline methods.

It improves the clarity and reliability of gradient-based explanations.

The method is simple, universal, and enhances interpretability.

Abstract

Gradient Smoothing is an efficient approach to reducing noise in gradient-based model explanation method. SmoothGrad adds Gaussian noise to mitigate much of these noise. However, the crucial hyper-parameter in this method, the variance $\sigma$ of Gaussian noise, is set manually or with heuristic approach. However, it results in the smoothed gradients still containing a certain amount of noise. In this paper, we aim to interpret SmoothGrad as a corollary of convolution, thereby re-understanding the gradient noise and the role of $\sigma$ from the perspective of confidence level. Furthermore, we propose an adaptive gradient smoothing method, AdaptGrad, based on these insights. Through comprehensive experiments, both qualitative and quantitative results demonstrate that AdaptGrad could effectively reduce almost all the noise in vanilla gradients compared with baselines methods. AdaptGrad is simple and universal, making it applicable for enhancing gradient-based interpretability methods for better visualization.