Win the Lottery Ticket via Fourier Analysis: Frequencies Guided Network Pruning

TL;DR

This paper introduces a Fourier analysis-based approach to guide network pruning, improving the effectiveness of magnitude-based pruning by analyzing frequency domain properties, and demonstrates its superiority through extensive experiments.

Contribution

It proposes a novel two-stage pruning method guided by Fourier analysis, enhancing network compression and performance recovery.

Findings

Fourier analysis explains MBP's generalization ability.

The proposed method outperforms traditional MBP algorithms.

Experiments on CIFAR datasets validate the approach.

Abstract

With the remarkable success of deep learning recently, efficient network compression algorithms are urgently demanded for releasing the potential computational power of edge devices, such as smartphones or tablets. However, optimal network pruning is a non-trivial task which mathematically is an NP-hard problem. Previous researchers explain training a pruned network as buying a lottery ticket. In this paper, we investigate the Magnitude-Based Pruning (MBP) scheme and analyze it from a novel perspective through Fourier analysis on the deep learning model to guide model designation. Besides explaining the generalization ability of MBP using Fourier transform, we also propose a novel two-stage pruning approach, where one stage is to obtain the topological structure of the pruned network and the other stage is to retrain the pruned network to recover the capacity using knowledge distillation from lower to higher on the frequency domain. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate the superiority of our novel Fourier analysis based MBP compared to other traditional MBP algorithms.