Scaling Deep Learning Research with Kubernetes on the NRP Nautilus HyperCluster

TL;DR

This paper demonstrates how the NRP Nautilus HyperCluster can automate and scale the training of numerous deep neural networks, significantly reducing training time for various applications in deep learning research.

Contribution

It introduces a scalable training framework using Kubernetes on the NRP Nautilus HyperCluster for multiple DNN applications, enhancing research efficiency.

Findings

234 models trained in 4,040 hours

Successful automation of DNN training workflows

Improved scalability for deep learning research

Abstract

Throughout the scientific computing space, deep learning algorithms have shown excellent performance in a wide range of applications. As these deep neural networks (DNNs) continue to mature, the necessary compute required to train them has continued to grow. Today, modern DNNs require millions of FLOPs and days to weeks of training to generate a well-trained model. The training times required for DNNs are oftentimes a bottleneck in DNN research for a variety of deep learning applications, and as such, accelerating and scaling DNN training enables more robust and accelerated research. To that end, in this work, we explore utilizing the NRP Nautilus HyperCluster to automate and scale deep learning model training for three separate applications of DNNs, including overhead object detection, burned area segmentation, and deforestation detection. In total, 234 deep neural models are trained on Nautilus, for a total time of 4,040 hours