Boost Neural Networks by Checkpoints

TL;DR

This paper introduces a boosting-based checkpoint ensemble method for deep neural networks that accelerates convergence, enhances diversity, and improves accuracy without increasing training costs, outperforming existing ensemble techniques.

Contribution

A novel boosting scheme for checkpoint ensembling that theoretically guarantees convergence and empirically improves accuracy and efficiency in deep neural network training.

Findings

Achieves 4.16% lower error on Cifar-100

Achieves 6.96% lower error on Tiny-ImageNet

Yields up to 5.02% higher accuracy on imbalanced datasets

Abstract

Training multiple deep neural networks (DNNs) and averaging their outputs is a simple way to improve the predictive performance. Nevertheless, the multiplied training cost prevents this ensemble method to be practical and efficient. Several recent works attempt to save and ensemble the checkpoints of DNNs, which only requires the same computational cost as training a single network. However, these methods suffer from either marginal accuracy improvements due to the low diversity of checkpoints or high risk of divergence due to the cyclical learning rates they adopted. In this paper, we propose a novel method to ensemble the checkpoints, where a boosting scheme is utilized to accelerate model convergence and maximize the checkpoint diversity. We theoretically prove that it converges by reducing exponential loss. The empirical evaluation also indicates our proposed ensemble outperforms single model and existing ensembles in terms of accuracy and efficiency. With the same training budget, our method achieves 4.16% lower error on Cifar-100 and 6.96% on Tiny-ImageNet with ResNet-110 architecture. Moreover, the adaptive sample weights in our method make it an effective solution to address the imbalanced class distribution. In the experiments, it yields up to 5.02% higher accuracy over single EfficientNet-B0 on the imbalanced datasets.