Evaluating Model Performance with Hard-Swish Activation Function Adjustments

TL;DR

This paper evaluates the impact of replacing ReLU with Hard-Swish activation functions in pre-trained models, demonstrating modest accuracy improvements across image datasets.

Contribution

It introduces an empirical comparison of activation functions, highlighting Hard-Swish as a beneficial alternative to ReLU in model fine-tuning.

Findings

2.06% accuracy increase on CIFAR-10

0.30% accuracy increase on ATLAS

Hard-Swish improves overall model accuracy

Abstract

In the field of pattern recognition, achieving high accuracy is essential. While training a model to recognize different complex images, it is vital to fine-tune the model to achieve the highest accuracy possible. One strategy for fine-tuning a model involves changing its activation function. Most pre-trained models use ReLU as their default activation function, but switching to a different activation function like Hard-Swish could be beneficial. This study evaluates the performance of models using ReLU, Swish and Hard-Swish activation functions across diverse image datasets. Our results show a 2.06% increase in accuracy for models on the CIFAR-10 dataset and a 0.30% increase in accuracy for models on the ATLAS dataset. Modifying the activation functions in architecture of pre-trained models lead to improved overall accuracy.