A Granger-Causal Perspective on Gradient Descent with Application to Pruning

TL;DR

This paper investigates the causal relationship between loss reduction and parameter updates in gradient descent, applying this perspective to pruning to improve neural network efficiency and interpretability.

Contribution

It introduces a causal framework for understanding gradient descent, explicitly linking loss reduction to parameter changes, and applies this to develop a novel pruning strategy.

Findings

Identification of a phase shift indicating optimal pruning levels

Pruning leads to flatter minima and increased accuracy

Causal perspective enhances control over training and pruning processes

Abstract

Stochastic Gradient Descent (SGD) is the main approach to optimizing neural networks. Several generalization properties of deep networks, such as convergence to a flatter minima, are believed to arise from SGD. This article explores the causality aspect of gradient descent. Specifically, we show that the gradient descent procedure has an implicit granger-causal relationship between the reduction in loss and a change in parameters. By suitable modifications, we make this causal relationship explicit. A causal approach to gradient descent has many significant applications which allow greater control. In this article, we illustrate the significance of the causal approach using the application of Pruning. The causal approach to pruning has several interesting properties - (i) We observe a phase shift as the percentage of pruned parameters increase. Such phase shift is indicative of an optimal pruning strategy. (ii) After pruning, we see that minima becomes "flatter", explaining the increase in accuracy after pruning weights.