Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning

TL;DR

This paper introduces Ensemble Knowledge Guidance (EKG), a novel method for sub-network search and fine-tuning in filter pruning that effectively predicts potential performance and reduces fine-tuning costs.

Contribution

It proposes EKG to evaluate potential performance via loss landscape fluctuation and guides both search and fine-tuning with minimal additional cost.

Findings

EKG effectively predicts potential performance.

Pruning ResNet-50 with 45% FLOPS reduction in 315 GPU hours.

EKG-guided pruning maintains performance without degradation.

Abstract

Conventional NAS-based pruning algorithms aim to find the sub-network with the best validation performance. However, validation performance does not successfully represent test performance, i.e., potential performance. Also, although fine-tuning the pruned network to restore the performance drop is an inevitable process, few studies have handled this issue. This paper provides a novel Ensemble Knowledge Guidance (EKG) to solve both problems at once. First, we experimentally prove that the fluctuation of loss landscape can be an effective metric to evaluate the potential performance. In order to search a sub-network with the smoothest loss landscape at a low cost, we employ EKG as a search reward. EKG utilized for the following search iteration is composed of the ensemble knowledge of interim sub-networks, i.e., the by-products of the sub-network evaluation. Next, we reuse EKG to provide a gentle and informative guidance to the pruned network while fine-tuning the pruned network. Since EKG is implemented as a memory bank in both phases, it requires a negligible cost. For example, when pruning and training ResNet-50, just 315 GPU hours are required to remove around 45.04% of FLOPS without any performance degradation, which can operate even on a low-spec workstation. the implemented code is available at https://github.com/sseung0703/EKG.