Multi-Resolution Model Fusion for Accelerating the Convolutional Neural Network Training

TL;DR

This paper introduces a multi-resolution model fusion method that accelerates CNN training by combining models trained on lower resolutions with high-resolution finetuning, significantly reducing training time without sacrificing accuracy.

Contribution

The paper proposes a novel multi-resolution fusion approach that speeds up CNN training by leveraging quick-to-generate lower-resolution models before final high-resolution refinement.

Findings

Training time reduced by up to 47% and 44% in two scientific applications.

Model accuracy remains unchanged despite reduced training time.

Method is effective for large high-dimensional data in scientific research.

Abstract

Neural networks are rapidly gaining popularity in scientific research, but training the models is often very time-consuming. Particularly when the training data samples are large high-dimensional arrays, efficient training methodologies that can reduce the computational costs are crucial. To reduce the training cost, we propose a Multi-Resolution Model Fusion (MRMF) method that combines models trained on reduced-resolution data and then refined with data in the original resolution. We demonstrate that these reduced-resolution models and datasets could be generated quickly. More importantly, the proposed approach reduces the training time by speeding up the model convergence in each fusion stage before switching to the final stage of finetuning with data in its original resolution. This strategy ensures the final model retains high-resolution insights while benefiting from the computational efficiency of lower-resolution training. Our experiment results demonstrate that the multi-resolution model fusion method can significantly reduce end-to-end training time while maintaining the same model accuracy. Evaluated using two real-world scientific applications, CosmoFlow and Neuron Inverter, the proposed method improves the training time by up to 47% and 44%, respectively, as compared to the original resolution training, while the model accuracy is not affected.