PFGE: Parsimonious Fast Geometric Ensembling of DNNs

TL;DR

PFGE introduces a memory-efficient ensemble method for deep neural networks that maintains high performance, leveraging stochastic weight averaging to reduce resource requirements significantly.

Contribution

The paper presents PFGE, a novel lightweight ensemble technique using stochastic weight averaging, achieving 5x memory savings over previous methods without losing accuracy.

Findings

PFGE achieves 5x memory efficiency compared to prior ensemble methods.

PFGE maintains comparable generalization performance to existing ensemble techniques.

Experimental validation on CIFAR-10, CIFAR-100, and ImageNet datasets demonstrates effectiveness.

Abstract

Ensemble methods are commonly used to enhance the generalization performance of machine learning models. However, they present a challenge in deep learning systems due to the high computational overhead required to train an ensemble of deep neural networks (DNNs). Recent advancements such as fast geometric ensembling (FGE) and snapshot ensembles have addressed this issue by training model ensembles in the same time as a single model. Nonetheless, these techniques still require additional memory for test-time inference compared to single-model-based methods. In this paper, we propose a new method called parsimonious FGE (PFGE), which employs a lightweight ensemble of higher-performing DNNs generated through successive stochastic weight averaging procedures. Our experimental results on CIFAR-{10,100} and ImageNet datasets across various modern DNN architectures demonstrate that PFGE achieves 5x memory efficiency compared to previous methods, without compromising on generalization performance. For those interested, our code is available at https://github.com/ZJLAB-AMMI/PFGE.