A Gradient Boosting Approach for Training Convolutional and Deep Neural Networks

TL;DR

This paper introduces gradient boosting-based procedures for training CNNs and DNNs, enhancing classification accuracy by iteratively adding and freezing dense layers, and demonstrating superior performance over standard models.

Contribution

The paper presents novel GB-CNN and GB-DNN training methods that improve deep neural network performance using gradient boosting and layer freezing techniques.

Findings

Superior classification accuracy on various datasets

Effective prevention of overfitting through layer freezing

Enhanced model fine-tuning capabilities

Abstract

Deep learning has revolutionized the computer vision and image classification domains. In this context Convolutional Neural Networks (CNNs) based architectures are the most widely applied models. In this article, we introduced two procedures for training Convolutional Neural Networks (CNNs) and Deep Neural Network based on Gradient Boosting (GB), namely GB-CNN and GB-DNN. These models are trained to fit the gradient of the loss function or pseudo-residuals of previous models. At each iteration, the proposed method adds one dense layer to an exact copy of the previous deep NN model. The weights of the dense layers trained on previous iterations are frozen to prevent over-fitting, permitting the model to fit the new dense as well as to fine-tune the convolutional layers (for GB-CNN) while still utilizing the information already learned. Through extensive experimentation on different 2D-image classification and tabular datasets, the presented models show superior performance in terms of classification accuracy with respect to standard CNN and Deep-NN with the same architectures.