Activation function cyclically switchable convolutional neural network model

TL;DR

This paper introduces a new CNN model that dynamically switches activation functions during training to improve performance.

Contribution

The novel AFCS-CNN model structure cyclically switches activation functions during training for better adaptability and performance.

Findings

AFCS-CNN achieved state-of-the-art results across multiple CNN models and datasets.

Ablation studies identified optimal hyperparameters and model configurations for the proposed structure.

The model structure is simple yet effective for various CNN applications.

Abstract

Neural networks are a state-of-the-art approach that performs well for many tasks. The activation function (AF) is an important hyperparameter that creates an output against the coming inputs to the neural network model. AF significantly affects the training and performance of the neural network model. Therefore, selecting the most optimal AF for processing input data in neural networks is important. Determining the optimal AF is often a difficult task. To overcome this difficulty, studies on trainable AFs have been carried out in the literature in recent years. This study presents a different approach apart from fixed or trainable AF approaches. For this purpose, the activation function cyclically switchable convolutional neural network (AFCS-CNN) model structure is proposed. The AFCS-CNN model structure does not use a fixed AF value during training. It is designed in a self-regulating model structure by switching the AF during model training. The proposed model structure is based on the logic of starting training with the most optimal AF selection among many AFs and cyclically selecting the next most optimal AF depending on the performance decrease during neural network training. Any convolutional neural network (CNN) model can be easily used in the proposed model structure. In this way, a simple but effective perspective has been presented. In this study, first, ablation studies have been carried out using the Cifar-10 dataset to determine the CNN models to be used in the AFCS-CNN model structure and the specific hyperparameters of the proposed model structure. After the models and hyperparameters were determined, expansion experiments were carried out using different datasets with the proposed model structure. The results showed that the AFCS-CNN model structure achieved state-of-the-art success in many CNN models and different datasets.

Figures

Examples of masked images from the V2 plant seedling dataset.

Examples of images from the APTOS 2019 Blindness Detection dataset.

Workflow of AFCS-CNN model structure.

Train-validation accuracy and loss graphs of each CNN model on the V2 plant seedling validation dataset.

Confusion matrix plots of each CNN model used in the AFCS-CNN model structure on the V2 plant seedling test dataset: (A) AFCS-CNN (VGG16), (B) AFCS-CNN (VGG19), (C) AFCS-CNN (DenseNet121), (D) AFCS-CNN (DenseNet169), (E) AFCS-CNN (EfficientNetV2B0), (F) AFCS-CNN (EfficientNetV2B1).

Train-validation accuracy and loss graphs of each CNN model on the APTOS 2019 Blindness Detection dataset.

Confusion matrix plots of each CNN model used in the AFCS-CNN model structure on the APTOS 2019 Blindness Detection test dataset: (A) AFCS-CNN (VGG16), (B) AFCS-CNN (VGG19), (C) AFCS-CNN (DenseNet121), (D) AFCS-CNN (DenseNet169), (E) AFCS-CNN (EfficientNetV2B0), (F) AFCS-CNN (EfficientNetV2B1).

Full text

Introduction

Neural networks are one of the modeling methods that play an important role in the fields of machine learning and deep learning (Goel, Goel & Kumar, 2023). The basic building blocks of these networks are artificial neurons inspired by biological nerve cells (Montesinos López, Montesinos López & Crossa, 2022; Liu et al., 2023). Artificial neurons that mimic biological neuron functions (Cao et al., 2022) apply a mathematical function to process the inputs they receive and produce output (Han et al., 2022; Gülmez, 2023; Yevick, 2024). In this process, the flow of information between neurons and the responses of neurons to inputs are largely regulated through activation function (AF) (Vijayaprabakaran & Sathiyamurthy, 2022). AF used in these processes is a critical element that determines how the neuron will respond to the signals it receives.

AFs directly affect the training and performance of the model as they provide an output based on the inputs in the neural network (Akgül, 2023). They are divided into two main categories: linear and nonlinear functions. While linear AFs enable the model to perform well on simpler problems, more complex functions are needed to model nonlinear relationships (Vargas et al., 2021). The task of AF is to learn abstract properties of data through nonlinear modification (Kalim, Chug & Singh, 2024). That is, to help learn nonlinear and complex mappings between inputs and their corresponding outputs (Sharma, Sharma & Athaiya, 2020; Subramanian et al., 2024). Recent research has found that AF plays an important role in introducing nonlinearity to improve the performance of deep learning networks (Rajanand & Singh, 2024). This nonlinear ability of AF has brought true artificial intelligence to deep neural networks (Wang, Ren & Wang, 2022).

Many AFs have been proposed throughout the history of machine learning. Some of the AF types commonly used in neural networks are: Binary Step, Linear, Sigmoid, Hyperbolic Tangent (Tanh), Rectified Linear Unit (ReLU), Leaky ReLU, Parametric RELU (PReLU), Exponential Linear Units (ELU), SELU, Softmax, Mish, Swish/Sigmoid Linear Unit (SiLU), Gaussian Error Linear Unit (GELU), Logish, Softplus and Softsign (Yuen et al., 2021; Wang, Ren & Wang, 2022; Kiaei et al., 2023; Kalim, Chug & Singh, 2024; Verma, Chug & Singh, 2024). AFs such as Sigmoid, Tanh, ReLU, and Leaky ReLU are the most widely used nonlinear functions in neural networks. Each function responds differently to inputs and offers specific advantages for specific problems. Sigmoid and Tanh AFs are both s-shaped and are used to classify objects where the output is limited to {0, 1} to {−1, 1} respectively (Verma, Chug & Singh, 2024). ReLU AF is used for quantification, classification, and reinforcement learning problems. Leaky ReLU, ELU, and Softplus AFs are part of the ReLU AF family. Leaky ReLU AF is a version of ReLU with a non-zero gradient to prevent the gradient from reaching zero. Elu and Softplus AFs solve the problem by creating smoothness and continuity in the environment. Newer AFs such as Mish, Swish/SiLU, and GELU have built-in regularization to prevent over-fitting of models (Yuen et al., 2021).

The purpose of machine learning algorithms is to determine the most optimal model to solve a specific problem. To determine the most optimal model, it is necessary to select the most optimal AF, like many parameters (Yuen et al., 2021). However, selecting the most optimal AF to train the neural network is not always an easy task (Kiaei et al., 2023). Since linear AFs limit the learning performance of deep learning models, nonlinear AFs are mostly preferred. Nonlinear AFs can be classified as fixed-parameter and trainable/adaptive AFs, depending on whether the AF parameter is fixed or modified during the training process of deep learning models (Kiliçarslan & Celik, 2024). Although the effectiveness of trainable or adaptive AFs has been examined in areas with abundant data, such as image classification problems, significant gaps remain in understanding their impact on classification accuracy and prediction uncertainty in environments characterized by limited data availability (Pourkamali-Anaraki et al., 2024).

Although fixed parameter AFs have been widely used in image classification models in previous studies, they have important shortcomings such as low classification accuracy. In recent years, adding parameters to AF to improve its performance has become a popular research topic. Excellent progress has been made with adaptive AFs obtained in this way (Jiang, Xie & Zhang, 2022). Many recent studies have been conducted on trainable or adaptive AF in the literature, especially in recent years (see ‘Related Works’). In these studies, a parametric variation of an existing AF was created by adding new parameters to commonly used AF types. Some of these created parametric AFs change the activation in the positive domain (Vargas et al., 2021). Another part focuses on either assigning a different slope to the negative domain of AF or converting the negative value to a positive value instead of zero (Jiang, Xie & Zhang, 2022). Although cyclic switching strategies such as cyclic learning rates (Smith, 2017) and cyclic precision (Fu et al., 2021) have been discussed during training, the cyclic AF switching strategy has not been discussed. In the studies, new AFs were generally proposed by adding an adaptive parameter to the AF, and the AF was used without a switch throughout the training. However, no study has been found on switching AF during training. In this study, unlike other studies, instead of an AF proposal, the ability to switch the AF with another AF during neural network training was adopted. The AFs to be used during training can be either fixed-parameter or adaptive. Thus, many AF switches are allowed during training (while training continues). In summary, it can be used in many AF neural networks thanks to the instantaneous AF switch while neural network training continues.

In this study, the activation function cyclically switchable convolutional neural network (AFCS-CNN) model structure was proposed, which allows cyclical switching of AF, which has an important place in neural networks. A first was achieved by proposing a new model from a unique perspective in the literature. Many experiments have been applied to determine the effectiveness of the proposed model. After a series of ablation experiments were carried out on the Cifar-10 dataset, expansion experiments were performed on the V2 Plant Seedling and APTOS 2019 Blindness Detection datasets, and the performance of the proposed model was measured. The highlight of this study to the literature can be summarized as follows:

1. A new model structure called AFCS-CNN has been proposed, which enables cyclical switching of AF.
2. Unlike the studies in the literature, instead of an AF proposal, the ability to switch the AF with another AF during neural network training has been adopted.
3. The concept of cyclic AF switching strategy during model training has been introduced.
4. A first was achieved by designing a model structure that had not been tried before, thanks to instant AF switches during neural network training.
5. The algorithm of the proposed model structure is designed to allow easy integration of all convolutional neural network (CNN) models into the structure.
6. Training with the proposed model structure has provided superior success in many problems compared to training with fixed AFs.
7. A state-of-the-art success has been achieved with the proposed model structure in plant seedling classification.

The remaining part of the study is organized as follows. Recent studies on AF development can be found in the ‘Related Works’, dataset preprocessing, training parameters, performance metrics and model structure are noted in ‘Materials and Methods’, performance and results are discussed in ‘Results and Discussions’, and the article concludes in the ‘Conclusion’.

Related works

In this section, recent studies on the development of trainable or adaptable AFs are comprehensively reviewed.

In a study on trainable or adaptive AFs, a trainable AF named Modified Mexican ReLU (MMeLU) was proposed to solve the complexity problem and improve the model performance. A fully Bayesian model is developed to automatically estimate both model weights and AF parameters from the training data. The proposed method is designed to learn the recommended AF weights and parameters directly from the data without any user configuration. The proposed method has been tested on various datasets covering both classification and regression tasks and implemented on various CNN architectures. The results showed that the proposed approach is useful in improving the model accuracy due to the proposed AF and Bayesian estimation of the parameters (Fakhfakh & Chaari, 2024). In another study, a new AF named ErfReLU based on error function (erf) and ReLU was developed. The performance of the proposed function is compared with nine different trainable AFs. CIFAR-10, MNIST, and FMNIST benchmark datasets were used to evaluate the effectiveness of AFs with MobileNet, VGG16, and ResNet models. The proposed AF has achieved better accuracy than other state-of-the-art adaptive AFs (Rajanand & Singh, 2024).

In another study, a novel trainable AF, adaptive piecewise approximated activation linear unit (APALU), was proposed to improve the learning performance of deep learning on a wide range of tasks. It has been stated that the proposed function exhibits significant improvements over AFs commonly used in image classification, anomaly detection, sign language recognition, and regression tasks (Subramanian et al., 2024). In another study, two trainable Gaussian-based AFs were proposed for sensor-based human activity recognition (HAR). Experiments have been conducted on Opportunity and UniMiB SHAR benchmark datasets with the proposed Four-Parameter Activation Gaussian Radial Basis Function (T4GRBF) and Weighted Gaussian Radial Basis Function (WGRBF). On both datasets, the results showed that Trainable Gaussian-based AFs fit the training data better and faster than standard AFs (Machacuay & Quinde, 2024). In another study, a generalized activation function called Generalized Exponential Parametric Activation Function (GEPAF) was proposed. In applications on real-world supply chain datasets, the proposed function outperformed popular AFs while demonstrating at least a 30% improvement in regression evaluation metrics and better loss reduction properties (Attarde & Sayyad, 2024).

In another study, PolySigmoid, PolySoftplus, PolyGeLU, PolySwish, and PolyMish polynomial versions of existing shape-shifting AFs such as Sigmoid, Softplus, GeLU, Swish, and Mish were proposed. As a result of the accuracy comparisons, it was determined that the performance of the polynomial versions of AFs was similar to or better than their counterparts (Herrera-Alcántara & Arellano-Balderas, 2024). Another study proposed an adaptive ReLU based on the genetic algorithm (GA) profile to improve the restrictive ability of the original ReLU and determine the best threshold value to allow adaptation-based neuron activation. As a result of experiments for breast cancer classification, it was found that the proposed method showed improved accuracy and improved classification performance from 95.0% to 98.5% compared to other well-known AFs (Razali et al., 2024).

Another study examined the effectiveness of trainable (non-periodic) AFs for neural implicit k-space (NIK) in the context of non-Cartesian Cardiac MRI. The proposed NIKs with trainable AFs are evaluated on 42 radially sampled datasets from six subjects and qualitatively and quantitatively outperform other state-of-the-art reconstruction methods including NIK with fixed periodic AFs (Haft et al., 2024). In another study, a new transformative adaptive activation function (TAAF) that allows any vertical and horizontal translation and scaling was investigated. It has been emphasized that TAAFs generalize more than 50 existing AFs and use similar concepts to more than 70 other AFs (Kunc, 2024).

In another work, a unified adaptive AF called adaptive activation (AdAct) was presented, which aims to improve neural network performance by adaptively selecting and combining the most suitable AFs for different tasks and datasets. The effectiveness of ReLU and its variants, including ELU, LReLU, PReLU, RReLU, and newer functions such as Swish and Mish, has been investigated by integrating them into the AdAct function. In the study where ConvNet variants were used on FMNIST, CIFAR10, SVHN, and FER datasets, AdAct showed better performance than other AFs (Maiti, 2024). In another study, in which the adaptive activation algorithm AdAct was proposed, it was emphasized that there were improvements in the performance of feedforward and convolutional neural networks (Rane et al., 2024). In another study, a new model, the expressive neural network (ENN), was presented in which nonlinear AFs were modeled using discrete cosine transform (DCT) and adapted using backpropagation during training. ENN has outperformed state-of-the-art non-adaptive AFs by adapting AFs for classification and regression problems (Martinez-Gost, Pérez-Neira & Lagunas, 2024).

In another study, a novel modulation window radial basis function neural network (MW-RBFNN) with a tunable AF was proposed. A raised cosine radial basis function (RC-RBF) was implemented as a shape-tunable AF of MW-RBFNN by adaptively modulating it by an exponential function. Simulation results of many different application cases have shown that this proposed MW-RBFNN is effective (Lin et al., 2024). In another study, a new adaptive deep neural networks (ADN) technique is proposed for partial differential equations (PDE) with corner singular solutions. Three different adaptive techniques were used for this, such as an adaptive loss function, an adaptive activation function, and an adaptive sampling strategy. The results showed that ADN achieved higher accuracy and faster convergence speed compared to some traditional DNN methods (Zeng, Liang & Zhang, 2024).

In a different study, it was studied that a customized adaptive activation function (AAF) can match the accuracy of a deep neural network (DNN). A field programmable gate arrays (FPGA) hardware implementation for a customized segmented spline curve neural network (SSCNN) structure is designed to replace the traditional fixed AF with an AAF. The proposed SSCNN implementation achieved similar accuracy using 40% less hardware resources and block RAM compared to DNN (Jiang et al., 2024). In a similar study, a local active memristor with a conductance function value range limited between 0 and 1 was designed to create an adaptive AF called memristive PReLU (MPReLU). A heterogeneous memristive Hopfield neural network with neurons using different activation functions was presented and a hardware implementation of this neural network with adaptive AF was designed. The experimental results largely coincided with those obtained using numerical simulations (Wang, Liang & Deng, 2024).

Materials and Methods

Dataset and preprocessing

In this study, the Cifar-10 (Krizhevsky, 2009) dataset was used to determine the CNN model to be used in the AFCS-CNN model structure and the special hyperparameters of AFCS-CNN, and V2 Plant Seedling (Kaggle, 2024; Giselsson et al., 2017) and APTOS 2019 Blindness Detection (APTOS 2019, 2024) datasets were used to test the model performance. Cifar-10 dataset consists of 10 classes and each of the 60,000 images in this dataset has dimensions of 32 × 32 × 3 pixels. The V2 Plant Seedling dataset consists of 12 classes representing common plant species in Danish agriculture, and each of the 5,539 images in this dataset has different pixel sizes. The Asia Pacific Tele-Ophthalmology Society 2019 Blindness Detection (APTOS 2019 Blindness Detection) dataset consists of five classes with an imbalanced distribution to classify diabetic retinopathy (DR), and each of the 3,662 images in this dataset has different pixel sizes. Table 1 lists the image types and numbers in each class in the datasets.

Table 1: Image types and numbers in each class in the datasets.

Since the images in the Cifar-10 dataset have the same pixel dimensions and there is no common distinctiveness in the objects in the images, they were not subjected to any preprocessing. Various preprocesses were applied to the images in the V2 Plant Seedling dataset to ensure smoother progress of the training process. In the V2 Plant Seedling dataset images, plant greens can be distinguished in color space because they are denser than other objects. Therefore, the images in this dataset were masked by applying color-based segmentation. The best color range to distinguish plant greens in the color space was determined between RGB: 27-45-45 and RGB: 100-255-255. After masking, the images in the V2 Plant Seedling dataset were set to 224 × 224 × 3 pixel dimensions. Since there is a class imbalance in the APTOS 2019 Blindness Detection dataset, no preprocessing other than dimension adjustment was performed to determine the behavior of the proposed model in an imbalanced dataset. Only the images in the dataset were set to 224 × 224 × 3 pixel dimensions. Figure 1 shows some examples obtained as a result of masking from images in the V2 Plant Seedling dataset, and Fig. 2 shows sample images of each class in the APTOS 2019 Blindness Detection dataset.

Examples of masked images from the V2 plant seedling dataset.

Examples of images from the APTOS 2019 Blindness Detection dataset.

Model training parameters and performance metrics

In the experiments conducted in this study, the Adamax optimization algorithm was used at its default values. The loss function was implemented as Categorical Crossentropy and the classification activation function was implemented as Softmax. To prevent the overfitting tendency of the models during training, the minimum level of loss value in the validation dataset was checked with the “patience” feature used in the “early stopping” process. Therefore, a fixed epoch value was not used. The batch size value was set to 128 for training with the Cifar-10 dataset and 16 for training with the V2 Plant Seedling and APTOS 2019 Blindness Detection datasets. The Cifar-10 dataset is divided into two datasets 80% training, and 20% testing, and the V2 Plant Seedling and APTOS 2019 Blindness Detection datasets are divided into three datasets 70% training, 15% validation, and 15% testing.

After the model is created, various evaluation metrics are needed to measure how its performance works (Sohan et al., 2019). Evaluation metrics mostly come from the confusion matrix (Zhang et al., 2016). Therefore, accuracy, loss, precision, recall, and F1-score metrics were examined to determine the classification success of the models.

AFCS-CNN model structure

AFCS-CNN model structure works on the principle of switching AF during model training. It suggests that AF can be switched cyclically depending on the performance decrease in the neural network. The workflow of the AFCS-CNN model structure is shown in Fig. 3, and the flowchart is shown in Algorithm 1.

Workflow of AFCS-CNN model structure.

Algorithm 1: AFCS-CNN algorithm.

The AFCS-CNN model structure, whose flow is shown in Fig. 3 and Algorithm 1, initially takes listAF, p0, p1, AFCSloop and minAFcount parameter values as input. In the neural network (ModelA), the best and worst performing AFs are determined by trying them among a series of AFs (listAF) determined as parameters at the input. Training starts with the best-performing AF, while the worst-performing AF is taken out from listAF. During training, whether the performance of the model on the validation dataset improves is checked with the patience (p0, p1) hyperparameter used in the early stopping process. In case the model performance does not improve during training, the best and the worst performing AFs are determined again among the remaining AFs. Training continues with the best-performing new AF. Take out process of the worst-performing AF from the system is continued cyclically until the minimum number of AFs (minAFcount) defined by the user at the beginning is reached. Training is finished when the number of AFCS cycles (AFCS_loop) defined at the beginning is reached.

The runtime of the AFCS-CNN model structure, whose flow is shown in Fig. 3 and Algorithm 1, is given in Eq. (1).

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$Runtim{e_{AFCS - CNN}} = R{t_{CNN}} + R{t_{ep}}\left( {{p_0} + \mathop \sum \limits_{i = 2}^{{\rm AFCS}\_{\rm loop}} {p_1}} \right)$$\end{document}

where RtCNN is the total runtime of any CNN model, Rtep is an epoch runtime and p0 and p1 are patience values.

According to Eq. (1), the runtime of the AFCS-CNN model is calculated by adding the cost that the p0 and p1 values will bring to the total runtime of the CNN model to be used in the AFCS-CNN model. The additional cost is found by multiplying the sum of the p0 value and the p1 value (as many as the number of cycles) by an epoch runtime of the CNN model. Considering the cases where the patience hyperparameter is used during model training, the additional runtime cost of the AFCS-CNN model can be ignored if the p0 and p1 parameters are given small values. For example, let’s assume that patience=10 is given in a CNN model that does not use the AFCS-CNN model structure and p0=6, p1=5 (AFCSloop=5) in the AFCS-CNN model. In this case, while the CNN model will run 10 epochs more, the AFCS-CNN model will run 6+5*4=26 epochs more. But in the AFCS-CNN model, when p0=3, p1=2 (AFCSloop=5) is given, the AFCS-CNN model will have run 3+2×4=11 epochs more. Or, when p0=2, p1=1 (AFCS_loop=5) is given in the AFCS-CNN model, the AFCS-CNN model will have run 2+1*4=6 epochs more. Therefore, it is recommended to give small values to the p0 and p1 parameters to obtain a competitive training cost with the AFCS-CNN model.

Results and discussion

In this study, ablation studies were performed using the Cifar-10 dataset to determine the CNN models to be used in the proposed AFCS-CNN model structure and the special hyperparameters of the proposed model structure, and expansion experiments were performed using the V2 Plant Seedling and APTOS 2019 Blindness Detection datasets to determine the performance of the proposed model structure.

Ablation studies

Ablation studies to determine the best CNN models and hyperparameters consist of three stages. The first stage includes determining the CNN models to be used in the AFCS-CNN model structure, the second stage includes determining the list_AF array and the p0 parameter value, and the third stage includes determining the p1 parameter value.

Stage 1: Determination of CNN models to be used in the AFCS-CNN model structure

To determine the CNN models to be used in the proposed AFCS-CNN model structure, VGG16 (Simonyan & Zisserman, 2014), ResNet50 (He et al., 2016), MobileNet (Howard, 2017), DenseNet121 (Huang et al., 2017), NASNetMobile (Zoph et al., 2018), EfficientNetV2B0 (Tan & Le, 2021), and ConvNeXtTiny (Liu et al., 2022) Keras models were tried on the Cifar-10 dataset. CNN models were trained for 20 epochs with pre-trained weights without partial layer freezing, and at default AF values. The numerical results obtained as a result of the training are given in Table 2.

Table 2: Numerical results obtained from model training in the Cifar-10 dataset.

According to the numerical results given in Table 2, the two best-performing models are VGG16 (84.52%) and DenseNet121 (82.93%), while the two worst-performing models are EfficientNetV2B0 (52.94%) and ConvNeXtTiny (10.00%). Since it would be appropriate to conduct ablation studies with the best-performing VGG16 and DenseNet121 models, these two models were preferred. It should be noted that in addition to these two best-performing models, the two worst-performing models (EfficientNetV2B0 and ConvNeXtTiny) were also used in the expansion experiments presented in the later sections of the study.

Stage 2: Determination of list_AF array and p0 parameter value

After determining the CNN models to be used in ablation studies, special hyperparameters of AFCS-CNN need to be defined. These hyperparameters can be given randomly or determined as a result of certain ablation studies. Here, a series of ablation studies were performed using the VGG16 and DenseNet121 models and the Cifar-10 dataset to determine the specific hyperparameters of AFCS-CNN.

Firstly, training was performed with VGG16 and DenseNet121 models to determine the listAF array and the p0 parameter value (epoch=20, patience=10). In addition to the ReLU, SiLU, and GELU AFs from the model default AF values presented in Table 2, the Tanh, ELU, SELU, and Mish AFs accepted in the literature were also preferred and the first version of the listAF array was created. The VGG16 and DenseNet121 models use ReLU AF by default. The default AF of VGG16 and DenseNet121 models were trained separately by replacing them with the AFs in the list_AF array (fixed AF throughout training). The experiments were repeated three times to determine the best AFs. The test loss values obtained as a result of experiments performed to measure the training performance of the models with each fixed AF are given in Table 3.

Table 3: Test loss values obtained as a result of experiments to determine the best AFs.

As a result of the experiments presented in Table 3, the Tanh function was removed from the AF list because learning did not occur with Tanh AF in the VGG16 model and the worst loss values were obtained with Tanh AF in the DenseNet121 model. Thus, the list_AF array was created as ReLU, ELU, SELU, Mish, SiLU, and GELU.

According to the experimental results given in Table 3, the consecutive loss increment numbers of the AFs were taken into account to determine the value of the p0 parameter. By examining the loss increase situations in Table 3, the consecutive loss increase numbers were determined and summarized numerically in Table 4.

Table 4: Consecutive loss increase numbers (determination of p0 value).

When Table 4 is examined, the overall average of consecutive loss increase numbers is calculated as 0.56 with the VGG16 model and 2.39 with the DenseNet121 model. Since training will be carried out with many models and datasets other than VGG16 and DenseNet121 models in the expansion experiments presented in the later sections of the study, the p0 value was determined by taking into account the DenseNet121 model, which has the highest average of consecutive loss increase numbers. Since the consecutive loss increase number = 2.39≈2, the p0 value is set to three. Thus, the loss will be allowed to increase two times in a row, and the training will be stopped at this stage when the loss value worsens for the 3rd time.

Stage 3: Determination of the p1 parameter value

After determining the list_AF array and p0 parameter values, to determine the p1 parameter value, the training was continued by selecting the three most successful AFs that reached the lowest loss value from the experiments performed with each model in Table 3 (epoch=20, patience=5). In this way, the behavior of the p1 parameter values according to different AF antecedents was investigated. Among the experiments performed with VGG16, the three most successful AFs that brought the model to the lowest loss value were SELU, ReLU, Mish in experiment 1, ELU, SELU, ReLU in experiment 2, and ReLU, GELU, ELU in experiment 3. Among the experiments performed with DenseNet121, the three most successful AFs that brought the model to the lowest loss value were GELU, ReLU, SiLU in experiment 1 and experiment 2, and ReLU, Mish, GELU in experiment 3. Training was continued on the models obtained with these AFs, and experiments were carried out to determine the best AF in the next step. The test loss values obtained as a result of the experiments are given in Table 5.

Table 5: Test loss values obtained as a result of experiments to determine the next best AFs.

According to the experimental results given in Table 5, the consecutive loss increment numbers of the AFs were taken into account to determine the value of the p1 parameter. By examining the loss increase situations in Table 5, the consecutive loss increase numbers were determined and summarized numerically in Table 6.

Table 6: Consecutive loss increase numbers (determination of p1 value).

When Table 6 is examined, the overall average of consecutive loss increase numbers is calculated as 0.02 with the VGG16 model and 1.38 with the DenseNet121 model. The p1 value was determined by taking into account the DenseNet121 model, which has the highest average of consecutive loss increase numbers. Since the consecutive loss increase number = 1.38≈1, the p1 value was set to two. Thus, the loss will be allowed to increase one time in a row, and the training will be stopped at this stage when the loss value worsens for the 2nd time.

In summary, as a result of ablation studies, the listAF array was created by taking the ReLU, ELU, SELU, Mish, SiLU, and GELU AFs where learning occurred. Thus, the initial values of the hyperparameter listAF, which holds the AF list to be applied to the CNN model to be used in the AFCS-CNN model structure were defined. The values of the patience hyperparameters used in the early stopping process were determined as three at the beginning and two in the next cycles. Patience values were kept small to prevent early overfitting of the model. Thus, the best patience values are defined as p0=3, p1=2. When the DenseNet121 model is used in the AFCS-CNN model structure, it is defined as AFCSloop=5 because the learning effect decreases after a certain cycle. The value of minAF_count was determined as 3, allowing the opportunity to select from at least three AFs during switches.

Expansion experiments

After ablation studies, VGG16, VGG19, DenseNet121, DenseNet169, EfficientNetV2B0, EfficientNetV2B1, ConvNeXtTiny, and ConvNeXtSmall CNN models were used in the proposed AFCS-CNN model structure to determine the performance success of the AFCS-CNN model structure. The hyperparameter values of the AFCS-CNN model structure are defined as determined in the previous section. So listAF = [ReLU, ELU, SELU, Mish, SiLU, GELU], p0=3, p1=2, AFCSloop=5, minAFcount=3 was set, and expansion experiments were performed on the V2 Plant Seedling and APTOS 2019 Blindness Detection datasets using VGG16, VGG19, DenseNet121, DenseNet169, EfficientNetV2B0, EfficientNetV2B1, ConvNeXtTiny, and ConvNeXtSmall CNN models in the AFCS-CNN model structure.

Expansion experiments with plant seedling dataset

To determine the performance success of the AFCS-CNN model structure on the V2 Plant Seedling dataset, experiments were performed using VGG16, VGG19, DenseNet121, DenseNet169, EfficientNetV2B0, EfficientNetV2B1, ConvNeXtTiny, and ConvNeXtSmall CNN models, respectively, in the AFCS-CNN model structure. In this way, the effect of AF switching during model training was examined. To compare the results, the default AF of each model was replaced by each AF in the list_AF array, and training was performed separately for each case (using the same AF throughout training). Experiments were performed using the train-validation dataset for the AFCS-CNN model structure and each fixed AF and the classification accuracy and loss values of the models in the V2 Plant Seedling validation dataset are given in Table 7.

Table 7: Success accuracy and loss values on the V2 plant seedling validation dataset.

When the validation accuracy and loss values given in Table 7 are examined, it is seen that using all CNN models discussed in the study in the AFCS-CNN model structure gives more successful results than not using them. The accuracy rates and loss values in the training performed using fixed AFs with CNN models without using the AFCS-CNN model structure were lower. Thanks to the AFCS-CNN model structure, it has been observed that switching AF provides a performance increase compared to fixing. While the most successful classification accuracy was achieved using the DenseNet121 model in the AFCS-CNN model (96.63%), no learning occurred in any of the training performed with ConvNeXt models. According to the validation success accuracies presented in Table 7, the accuracy and loss graphs of the three CNN models with the best performing AF and each CNN model used in the AFCS-CNN model structure are shown comparatively in Fig. 4.

Train-validation accuracy and loss graphs of each CNN model on the V2 plant seedling validation dataset.

In Fig. 4, graphs of the training performed with the most successful AFs that were left fixed throughout the training of each CNN model and the training performed using each CNN model in the AFCS-CNN model structure are shown together. As can be seen in the figure, the proposed AFCS-CNN model structure performs the training process in an up-and-down structure. The ups and downs in training the AFCS-CNN model structure are due to the switching of AF. For example, in the case where the VGG16 model was used in the AFCS-CNN model structure, the AFCS-CNN model structure started training with GELU AF, which performs best among the values in the listAF array. Since the validation loss value increased three times (p0 control) in a row after the 13th epoch, AF=Mish was made in this epoch and the training was continued from the 13th epoch. Since the worst performing AF was SELU, this AF was taken out from the listAF array to not be used in the next switch. Again, after the 15th epoch, when the validation loss value increased two times (p1 control), AF=SiLU was made, and the training was continued from the 15th epoch. At this stage, ELU AF was taken out from the listAF array. Then, in the 17th epoch, AF=Mish was made, and ReLU AF was taken out from the listAF array. In the 20th epoch, AF=GELU was made, and the cycle was finished in the 24th epoch. As seen in Fig. 4, in the training performed using only the VGG16 model and fixed AFs without using the AFCS-CNN model structure, it was observed that the difference between the train-validation graph widened after a certain epoch and overfitting occurred. It was determined that when the VGG16 model is used in the AFCS-CNN model structure, overfitting is prevented, and learning occurs in fewer epochs. Thus, in the training performed with the VGG16 model, it was observed that the AF switch during the training process with the AFCS-CNN model structure had a positive effect on the model training process.

In addition, it was observed that when other CNN models discussed in the study were used in the AFCS-CNN model structure, different AF switches occurred at different epochs and the AF switch had a positive effect on the model training process. When the train-validation accuracy and loss graphs shown in Fig. 4 are examined, the models made five AF switches depending on the number of cycles defined at the beginning. In the training of the CNN models used in the AFCS-CNN model structure on the V2 Plant Seedling dataset, different AF switches were made at different epochs, and the details are summarized in Table 8.

Table 8: AF switches in the training of the CNN models used in the AFCS-CNN model structure on the V2 plant seedling dataset.

After the train-validation process, the final performance of the models was tested using the V2 Plant Seedling test dataset, which the network had never seen, and the numerical results are given in Table 9. Additionally, Fig. 5 shows the confusion matrix results obtained from the test dataset with each CNN model used in the AFCS-CNN model structure.

Table 9: Numerical results obtained from models using the V2 plant seedling test dataset.

Confusion matrix plots of each CNN model used in the AFCS-CNN model structure on the V2 plant seedling test dataset: (A) AFCS-CNN (VGG16), (B) AFCS-CNN (VGG19), (C) AFCS-CNN (DenseNet121), (D) AFCS-CNN (DenseNet169), (E) AFCS-CNN (EfficientNetV2B0), (F) AFCS-CNN (EfficientNetV2B1).

According to the results given in Table 9 and Fig. 5, the most successful classification accuracy in the V2 Plant Seedling test dataset was achieved by using the DenseNet121 model in the AFCS-CNN model. With the AFCS-CNN (DenseNet121) model, 97.11% accuracy and 0.0904 loss values were obtained in the test dataset. It has been observed that using all CNN models discussed in the study in the AFCS-CNN model structure gives more successful results than using fixed AF in CNN models. Thus, it has been demonstrated that the AFCS-CNN model structure provides an important performance increase in all CNN models discussed in the study.

Expansion experiments with APTOS 2019 Blindness Detection dataset

To determine the performance success of the AFCS-CNN model structure on the APTOS 2019 Blindness Detection dataset, experiments were performed using VGG16, VGG19, DenseNet121, DenseNet169, EfficientNetV2B0, EfficientNetV2B1, ConvNeXtTiny, and ConvNeXtSmall CNN models, respectively, in the AFCS-CNN model structure. In this way, the effect of AF switching during model training was examined using a different dataset. To compare the results, the default AF of each model was replaced by each AF in the list_AF array, and training was performed separately for each case (using the same AF throughout training). Experiments were performed using the train-validation dataset for the AFCS-CNN model structure and each fixed AF and the classification accuracy and loss values of the models in the APTOS 2019 Blindness Detection validation dataset are given in Table 10.

Table 10: Success accuracy and loss values on the APTOS 2019 Blindness Detection Validation dataset.

When the validation accuracy and loss values given in Table 10 are examined, it is seen that using all CNN models, except VGG models, in the AFCS-CNN model structure gives more successful results than not using them. The accuracy rates and loss values in the training performed using fixed AFs with CNN models without using the AFCS-CNN model structure were lower. Thanks to the AFCS-CNN model structure, it has been observed that switching AF provides a performance increase compared to fixing. While the most successful classification accuracy was achieved using the DenseNet169 model in the AFCS-CNN model (83.45%), no learning occurred in any of the training performed with ConvNeXt models. According to the validation success accuracies presented in Table 10, the accuracy and loss graphs of the three CNN models with the best performing AF and each CNN model used in the AFCS-CNN model structure are shown comparatively in Fig. 6.

Train-validation accuracy and loss graphs of each CNN model on the APTOS 2019 Blindness Detection dataset.

In Fig. 6, graphs of the training performed with the most successful AFs that were left fixed throughout the training of each CNN model and the training performed using each CNN model in the AFCS-CNN model structure are shown together. As can be seen in the figure, similar to the previous analyses, the proposed AFCS-CNN model structure performs the training process in an up-and-down structure. The ups and downs in training the AFCS-CNN model structure are due to the switching of AF. For example, in the case where the VGG16 model was used in the AFCS-CNN model structure, the AFCS-CNN model structure started training with ReLU AF, which performs best among the values in the listAF array. Since the validation loss value increased three times (p0 control) in a row after the 11th epoch, AF=SiLU was made in this epoch and the training was continued from the 11th epoch. Since the worst performing AF was ELU, this AF was taken out from the listAF array to not be used in the next switch. Again, after the 16th epoch, when the validation loss value increased two times (p1 control), AF=ReLU was made, and the training was continued from the 16th epoch. At this stage, SELU AF was taken out from the listAF array. Then, in the 28th epoch, AF=Mish was made, and GELU AF was taken out from the listAF array. In the 32nd epoch, AF=SiLU was made, and the cycle was finished in the 35th epoch. As seen in Fig. 6, in the training performed using only the VGG16 model and fixed AFs without using the AFCS-CNN model structure, it was observed that the difference between the train-validation graph widened after a certain epoch and overfitting occurred. It was determined that when the VGG16 model is used in the AFCS-CNN model structure, overfitting is prevented, and learning occurs in fewer epochs. Thus, in the training performed with the VGG16 model, it was observed that the AF switch during the training process with the AFCS-CNN model structure had a positive effect on the model training process.

In addition, it was observed that when other CNN models discussed in the study were used in the AFCS-CNN model structure, different AF switches occurred at different epochs and the AF switch had a positive effect on the model training process. When the train-validation accuracy and loss graphs shown in Fig. 6 are examined, the models made five AF switches depending on the number of cycles defined at the beginning. In the training of the CNN models used in the AFCS-CNN model structure on the APTOS 2019 Blindness Detection dataset, different AF switches were made at different epochs, and the details are summarized in Table 11.

Table 11: AF switches in the training of CNN models used in the AFCS-CNN model structure on the APTOS 2019 Blindness Detection dataset.

After the train-validation process, the final performance of the models was tested using the APTOS 2019 Blindness Detection test dataset, which the network had never seen, and the numerical results are given in Table 12. Additionally, Fig. 7 shows the confusion matrix results obtained from the test dataset with each CNN model used in the AFCS-CNN model structure.

Table 12: Numerical results obtained from models using the APTOS 2019 Blindness Detection test dataset.

Confusion matrix plots of each CNN model used in the AFCS-CNN model structure on the APTOS 2019 Blindness Detection test dataset: (A) AFCS-CNN (VGG16), (B) AFCS-CNN (VGG19), (C) AFCS-CNN (DenseNet121), (D) AFCS-CNN (DenseNet169), (E) AFCS-CNN (EfficientNetV2B0), (F) AFCS-CNN (EfficientNetV2B1).

According to the results given in Table 12 and Fig. 7, the most successful classification accuracy in the APTOS 2019 Blindness Detection test dataset was achieved by using the DenseNet121 model in the AFCS-CNN model. With the AFCS-CNN (DenseNet121) model, 83.27% accuracy and 0.5704 loss values were obtained in the test dataset. It has been observed that using all CNN models discussed in the study in the AFCS-CNN model structure gives more successful results than using fixed AF in CNN models. Thus, it has been demonstrated that the AFCS-CNN model structure provides an important performance increase in all CNN models discussed in the study.

Discussion

To determine the performance of the AFCS-CNN model structure, the experimental results performed with many CNN models and many datasets discussed in this study have shown that the proposed AFCS-CNN model structure provides an important performance increase. Experiments with different datasets have demonstrated that using CNN models in the AFCS-CNN model structure is more successful than using fixed AF in CNN models. However, using a CNN model in the AFCS-CNN model structure has an additional runtime cost compared to not using it. Since the hyperparameter values of the AFCS-CNN model are p0=3, p1=2 and AFCS_loop=5, the additional running time of the model will be 11 epochs according to the formula given in Eq. (1). Considering that the patience value in a CNN model is usually given as 5 or 10, the additional runtime cost between 1 and 6 epochs brought by the AFCS-CNN model can be ignored when the performance achieved is taken into account.

Among the datasets discussed in the study, only the V2 Plant Seedling dataset was compared with current works because various preprocessing was applied to this dataset. For this purpose, to compare the plant seedling classification success of the AFCS-CNN model structure, recent works using the V2 Plant Seedling dataset were examined in the literature and the results are given in Table 13.

Table 13: Comparison of the proposed work with recent works performed on the V2 plant seedling dataset.

When Table 13 is examined, the proposed AFCS-CNN (DenseNet121) model is more successful in plant seedling classification than all other works except the work done by Aliouat, Badis & Bouchiba (2023). Data augmentation was done in the work by Aliouat, Badis & Bouchiba (2023). However, data augmentation was not used in our work. In addition, high success was achieved with the proposed AFCS-CNN models with fewer iterations. Therefore, it was observed that the proposed AFCS-CNN model structure is more successful and competitive with other methods in plant seedling classification. Thus, the AFCS-CNN model structure has achieved state-of-the-art success.

Conclusion

In this study, a new model structure called AFCS-CNN was proposed, which allows cyclical switching of the AF during training, depending on the performance decrease in the neural network, and can be applied to any CNN model. Many experiments were applied to determine the hyperparameters of the proposed model and measure its performance. After a series of ablation studies using the Cifar-10 dataset, hyperparameter values of the AFCS-CNN model structure were determined. In addition, CNN models were chosen to be used in ablation studies. After ablation studies, expansion experiments were performed with V2 Plant Seedling and APTOS 2019 Blindness Detection datasets using VGG, DenseNet, EfficientNetV2, and ConvNeXt versions in the AFCS-CNN model structure.

As a result of expansion experiments, more successful results were obtained with the AFCS-CNN model structure than all the CNN models discussed in the study in both V2 Plant Seedling and APTOS 2019 Blindness Detection datasets. In all experiments, the most successful results were achieved with the AFCS-CNN (DenseNet121) model. Especially in plant seedling classification, a high-test success accuracy of 97.11% was achieved with the AFCS-CNN (DenseNet121) model with few iterations. In addition, thanks to the use of all other CNN models discussed in the study in the AFCS-CNN model structure, higher test success accuracies were achieved in both datasets. Moreover, the proposed AFCS-CNN model structure was compared with similar works in the literature on plant seedling classification and was more successful than the state-of-the-art works in terms of accuracy and number of iterations. It is planned to use different CNN models in the AFCS-CNN model structure in future studies. Moreover, the applicability of the proposed method in different areas will be tested.

Supplemental Information

10.7717/peerj-cs.2756/supp-1Supplemental Information 1Highlights.

10.7717/peerj-cs.2756/supp-2Supplemental Information 2Ablation studies performed with Cifar-10 dataset.

10.7717/peerj-cs.2756/supp-3Supplemental Information 3Trainings performed on V2 Plant Seedling dataset with AFCS-CNN (VGG16) model.

10.7717/peerj-cs.2756/supp-4Supplemental Information 4Trainings performed on V2 Plant Seedling dataset with AFCS-CNN (VGG19) model.

10.7717/peerj-cs.2756/supp-5Supplemental Information 5Trainings performed on V2 Plant Seedling dataset with AFCS-CNN (DenseNet121) model.

10.7717/peerj-cs.2756/supp-6Supplemental Information 6Trainings performed on V2 Plant Seedling dataset with AFCS-CNN (DenseNet169) model.

10.7717/peerj-cs.2756/supp-7Supplemental Information 7Trainings performed on V2 Plant Seedling dataset with AFCS-CNN (EfficientNetV2B0) model.

10.7717/peerj-cs.2756/supp-8Supplemental Information 8Trainings performed on V2 Plant Seedling dataset with AFCS-CNN (EfficientNetV2B1) model.

10.7717/peerj-cs.2756/supp-9Supplemental Information 9Trainings performed on V2 Plant Seedling dataset with AFCS-CNN (ConvNeXtTiny) model.

10.7717/peerj-cs.2756/supp-10Supplemental Information 10Trainings performed on V2 Plant Seedling dataset with AFCS-CNN (ConvNeXtSmall) model.

10.7717/peerj-cs.2756/supp-11Supplemental Information 11Trainings performed on APTOS 2019 Blindness Detection dataset with AFCS-CNN (VGG16) model.

10.7717/peerj-cs.2756/supp-12Supplemental Information 12Trainings performed on APTOS 2019 Blindness Detection dataset with AFCS-CNN (VGG19) model.

10.7717/peerj-cs.2756/supp-13Supplemental Information 13Trainings performed on APTOS 2019 Blindness Detection dataset with AFCS-CNN (DenseNet121) model.

10.7717/peerj-cs.2756/supp-14Supplemental Information 14Trainings performed on APTOS 2019 Blindness Detection dataset with AFCS-CNN (DenseNet169) model.

10.7717/peerj-cs.2756/supp-15Supplemental Information 15Trainings performed on APTOS 2019 Blindness Detection dataset with AFCS-CNN (EfficientNetV2B0) model.

10.7717/peerj-cs.2756/supp-16Supplemental Information 16Trainings performed on APTOS 2019 Blindness Detection dataset with AFCS-CNN (EfficientNetV2B1) model.

10.7717/peerj-cs.2756/supp-17Supplemental Information 17Trainings performed on APTOS 2019 Blindness Detection dataset with AFCS-CNN (ConvNeXtTiny) model.

10.7717/peerj-cs.2756/supp-18Supplemental Information 18Trainings performed on APTOS 2019 Blindness Detection dataset with AFCS-CNN (ConvNeXtSmall) model.