Scaling Shared-Memory Data Structures as Distributed Global-View Data Structures in the Partitioned Global Address Space model

TL;DR

This paper introduces a method to adapt shared-memory data structures for scalable use in distributed memory systems within the PGAS model, exemplified by the DIHT, which significantly improves performance.

Contribution

It presents the Distributed Interlocked Hash Table (DIHT), a novel global-view distributed map structure inspired by IHT, optimized for PGAS systems.

Findings

DIHT achieves up to 110x performance over standard library implementations.

The approach effectively scales shared-memory data structures in distributed environments.

Demonstrates practical benefits on a 64-node, 44-core per node cluster.

Abstract

The Partitioned Global Address Space (PGAS), a memory model in which the global address space is explicitly partitioned across compute nodes in a cluster, strives to bridge the gap between shared-memory and distributed-memory programming. To further bridge this gap, there has been an adoption of global-view distributed data structures, such as 'global arrays' or 'distributed arrays'. This work demonstrates how shared-memory data structures can be modified to scale in distributed memory. Presented in this work is the Distributed Interlocked Hash Table (DIHT), a global-view distributed map data structure inpired by the Interlocked Hash Table (IHT). At 64 nodes with 44 cores per node, DIHT provides upto 110x the performance of the Chapel standard-library HashedDist.