Extending a serial 3D two-phase CFD code to parallel execution over MPI by using the PETSc library for domain decomposition

TL;DR

This paper demonstrates how to convert a serial 3D two-phase CFD code into a parallel version using MPI and PETSc, enabling large-scale simulations of oil/water separation with minimal code changes.

Contribution

The work introduces a method to parallelize an existing serial CFD code with MPI and PETSc, focusing on domain decomposition with limited code modifications.

Findings

Achieved scalable parallel performance for the CFD code

Enabled large-scale simulations relevant to oil/water separation

Validated strong and weak scaling of the parallel code

Abstract

To leverage the last two decades' transition in High-Performance Computing (HPC) towards clusters of compute nodes bound together with fast interconnects, a modern scalable CFD code must be able to efficiently distribute work amongst several nodes using the Message Passing Interface (MPI). MPI can enable very large simulations running on very large clusters, but it is necessary that the bulk of the CFD code be written with MPI in mind, an obstacle to parallelizing an existing serial code. In this work we present the results of extending an existing two-phase 3D Navier-Stokes solver, which was completely serial, to a parallel execution model using MPI. The 3D Navier-Stokes equations for two immiscible incompressible fluids are solved by the continuum surface force method, while the location of the interface is determined by the level-set method. We employ the Portable Extensible Toolkit for Scientific Computing (PETSc) for domain decomposition (DD) in a framework where only a fraction of the code needs to be altered. We study the strong and weak scaling of the resulting code. Cases are studied that are relevant to the fundamental understanding of oil/water separation in electrocoalescers.