Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis

TL;DR

This paper explores the connection between Iterative Magnitude Pruning in neural networks and Renormalisation Group theory, investigating the universality of winning tickets within the framework of the Lottery Ticket Hypothesis.

Contribution

It introduces a novel perspective by linking IMP to RG theory and examines the universality of winning tickets across different problems.

Findings

IMP can be interpreted through the lens of RG theory.

Winning tickets exhibit universality across similar tasks.

The study provides a theoretical foundation for understanding pruning in DNNs.

Abstract

This thesis delves into the intricate world of Deep Neural Networks (DNNs), focusing on the exciting concept of the Lottery Ticket Hypothesis (LTH). The LTH posits that within extensive DNNs, smaller, trainable subnetworks termed "winning tickets", can achieve performance comparable to the full model. A key process in LTH, Iterative Magnitude Pruning (IMP), incrementally eliminates minimal weights, emulating stepwise learning in DNNs. Once we identify these winning tickets, we further investigate their "universality". In other words, we check if a winning ticket that works well for one specific problem could also work well for other, similar problems. We also bridge the divide between the IMP and the Renormalisation Group (RG) theory in physics, promoting a more rigorous understanding of IMP.