Supporting 64-bit global indices in Epetra and other Trilinos packages -- Techniques used and lessons learned

TL;DR

This paper describes the addition of 64-bit global index support in the Epetra package of Trilinos, enabling larger problem sizes and discussing the technical and design lessons learned during implementation.

Contribution

It introduces optional 64-bit global indices in Epetra and shares insights on modifying a mature software package for enhanced scalability.

Findings

Enabled support for larger problem sizes exceeding two billion DOFs.

Maintained memory efficiency and performance with 32-bit local indices.

Provided detailed documentation and lessons learned for similar software modifications.

Abstract

The Trilinos Project is an effort to facilitate the design, development, integration and ongoing support of mathematical software libraries within an object-oriented framework. It is intended for large-scale, complex multiphysics engineering and scientific applications. Epetra is one of its basic packages. It provides serial and parallel linear algebra capabilities. Before Trilinos version 11.0, released in 2012, Epetra used the C++ int data-type for storing global and local indices for degrees of freedom (DOFs). Since int is typically 32-bit, this limited the largest problem size to be smaller than approximately two billion DOFs. This was true even if a distributed memory machine could handle larger problems. We have added optional support for C++ long long data-type, which is at least 64-bit wide, for global indices. To save memory, maintain the speed of memory-bound operations, and reduce further changes to the code, the local indices are still 32-bit. We document the changes required to achieve this feature and how the new functionality can be used. We also report on the lessons learned in modifying a mature and popular package from various perspectives -- design goals, backward compatibility, engineering decisions, C++ language features, effects on existing users and other packages, and build integration.