Learning Activation Functions for Sparse Neural Networks

TL;DR

This paper introduces SAFS, a method for optimizing activation functions and hyperparameters in sparse neural networks, significantly improving accuracy at high pruning ratios across multiple models and datasets.

Contribution

It proposes a novel combined approach for tuning activation functions and hyperparameters specifically for sparse neural networks, addressing accuracy drops.

Findings

SAFS improves accuracy by up to 15.53% on LeNet-5.

SAFS achieves up to 8.88% improvement on VGG-16.

SAFS enhances ResNet-18 accuracy by 6.33% at high pruning ratios.

Abstract

Sparse Neural Networks (SNNs) can potentially demonstrate similar performance to their dense counterparts while saving significant energy and memory at inference. However, the accuracy drop incurred by SNNs, especially at high pruning ratios, can be an issue in critical deployment conditions. While recent works mitigate this issue through sophisticated pruning techniques, we shift our focus to an overlooked factor: hyperparameters and activation functions. Our analyses have shown that the accuracy drop can additionally be attributed to (i) Using ReLU as the default choice for activation functions unanimously, and (ii) Fine-tuning SNNs with the same hyperparameters as dense counterparts. Thus, we focus on learning a novel way to tune activation functions for sparse networks and combining these with a separate hyperparameter optimization (HPO) regime for sparse networks. By conducting experiments on popular DNN models (LeNet-5, VGG-16, ResNet-18, and EfficientNet-B0) trained on MNIST, CIFAR-10, and ImageNet-16 datasets, we show that the novel combination of these two approaches, dubbed Sparse Activation Function Search, short: SAFS, results in up to 15.53%, 8.88%, and 6.33% absolute improvement in the accuracy for LeNet-5, VGG-16, and ResNet-18 over the default training protocols, especially at high pruning ratios. Our code can be found at https://github.com/automl/SAFS