Conduit: A C++ Library for Best-effort High Performance Computing

TL;DR

Conduit is a C++ library designed to facilitate best-effort high-performance computing by relaxing synchronization, which improves scalability and efficiency in distributed systems.

Contribution

It introduces a novel C++ library providing a best-effort interface for inter-thread and inter-process communication in high-performance computing.

Findings

Improves scaling efficiency in distributed systems.

Enhances solution quality in communication-intensive tasks.

Demonstrates benefits in multi-node environments.

Abstract

Developing software to effectively take advantage of growth in parallel and distributed processing capacity poses significant challenges. Traditional programming techniques allow a user to assume that execution, message passing, and memory are always kept synchronized. However, maintaining this consistency becomes increasingly costly at scale. One proposed strategy is "best-effort computing", which relaxes synchronization and hardware reliability requirements, accepting nondeterminism in exchange for efficiency. Although many programming languages and frameworks aim to facilitate software development for high performance applications, existing tools do not directly provide a prepackaged best-effort interface. The Conduit C++ Library aims to provide such an interface for convenient implementation of software that uses best-effort inter-thread and inter-process communication. Here, we describe the motivation, objectives, design, and implementation of the library. Benchmarks on a communication-intensive graph coloring problem and a compute-intensive digital evolution simulation show that Conduit's best-effort model can improve scaling efficiency and solution quality, particularly in a distributed, multi-node context.