Distributed Deep Learning for Precipitation Nowcasting

TL;DR

This paper demonstrates that distributed training of CNNs across multiple GPUs drastically reduces training time for precipitation nowcasting models, enabling faster development and iteration of weather forecasting neural networks.

Contribution

It introduces a data-parallel distributed training approach for CNNs in precipitation nowcasting, significantly reducing training time from 59 hours to just over 1 hour.

Findings

Training time reduced from 59 hours to over 1 hour

Distributed training enables faster model development

Facilitates future advancements in nowcasting models

Abstract

Effective training of Deep Neural Networks requires massive amounts of data and compute. As a result, longer times are needed to train complex models requiring large datasets, which can severely limit research on model development and the exploitation of all available data. In this paper, this problem is investigated in the context of precipitation nowcasting, a term used to describe highly detailed short-term forecasts of precipitation and other hazardous weather. Convolutional Neural Networks (CNNs) are a powerful class of models that are well-suited for this task; however, the high resolution input weather imagery combined with model complexity required to process this data makes training CNNs to solve this task time consuming. To address this issue, a data-parallel model is implemented where a CNN is replicated across multiple compute nodes and the training batches are distributed across multiple nodes. By leveraging multiple GPUs, we show that the training time for a given nowcasting model architecture can be reduced from 59 hours to just over 1 hour. This will allow for faster iterations for improving CNN architectures and will facilitate future advancement in the area of nowcasting.