DevOps and Microservices in Scientific System development

TL;DR

This paper explores how integrating DevOps and microservices with agile methods can improve scientific system development by enabling scientists to focus on their expertise while ensuring high-quality software engineering.

Contribution

It presents a multi-year industry research experience demonstrating the benefits of applying microservices and DevOps in scientific software development.

Findings

Microservices enable scientists to develop prototypes more easily.

DevOps practices improve integration and deployment processes.

Enhanced collaboration between scientists and software engineers.

Abstract

There is a gap in scientific information systems development concerning modern software engineering and scientific computing. Historically, software engineering methodologies have been perceived as an unwanted accidental complexity to computational scientists in their scientific systems development. More recent trends, like the end of Moore's law and the subsequent diversification of hardware platforms, combined with the increasing multidisciplinarity of science itself have exacerbated the problem because self-taught "end user developers" are not familiar with the disciplines needed to tackle this increased complexity. On a more positive note, agile programming methods have approached software development practices to the way scientific software is produced. In this work, we present the experience of a multi-year industry research project where agile methods, microservices and DevOps were applied. Our goal is to validate the hypothesis that the use of microservices would allow computational scientists to work in the more minimalistic prototype-oriented way that they prefer while the software engineering team would handle the integration. Hence, scientific multidisciplinary systems would gain in a twofold way: (i) Subject Matter Experts(SME) use their preferable tools to develop the specific scientific part of the system; (ii) software engineers provide the high quality software code for the system delivery.