Efficient Per-Example Gradient Computations

TL;DR

This paper introduces an efficient method to compute the gradient norm for each example in a neural network, enabling faster analysis of model sensitivities and training dynamics.

Contribution

It presents a novel technique that significantly reduces the computational cost of per-example gradient norm calculations in neural networks.

Findings

Reduces computation time for per-example gradient norms

Enables detailed analysis of training dynamics

Improves understanding of model sensitivities

Abstract

This technical report describes an efficient technique for computing the norm of the gradient of the loss function for a neural network with respect to its parameters. This gradient norm can be computed efficiently for every example.