Toward a Standard Interface for User-Defined Scheduling in OpenMP

TL;DR

This paper proposes extending OpenMP to support user-defined scheduling strategies, enabling more flexible and application-specific parallel loop performance improvements.

Contribution

It introduces two candidate interfaces for user-defined scheduling in OpenMP, facilitating standard compliance and customization.

Findings

Two competing interfaces are proposed for user-defined scheduling.

Conceptual comparison across C, C++, and Fortran.

Framework serves as a basis for discussion and prototyping.

Abstract

Parallel loops are an important part of OpenMP programs. Efficient scheduling of parallel loops can improve performance of the programs. The current OpenMP specification only offers three options for loop scheduling, which are insufficient in certain instances. Given the large number of other possible scheduling strategies, it is infeasible to standardize each one. A more viable approach is to extend the OpenMP standard to allow for users to define loop scheduling strategies. The approach will enable standard-compliant application-specific scheduling. This work analyzes the principal components required by user-defined scheduling and proposes two competing interfaces as candidates for the OpenMP standard. We conceptually compare the two proposed interfaces with respect to the three host languages of OpenMP, i.e., C, C++, and Fortran. These interfaces serve the OpenMP community as a basis for discussion and prototype implementation for user-defined scheduling.

Tables

⬇ #pragma omp parallel for for (i=0;i<iMax;i++) { ... LOOP BODY ... }

$\to$

⬇ #pragma omp parallel { init(...); #pragma omp barrier while(!done){ for (each item in dequeue(...)) ... LOOP BODY ... } finalize(...); }

Full text

Keywords: OpenMP, multithreaded applications, shared-memory programming, multicore, loop scheduling, self-scheduling, user-defined loop scheduling, dynamic load balancing, high performance computing.

1 Introduction

OpenMP  [9] is the industry and academic standard for parallel programming on shared memory platforms. Loop-level parallelism is a very important part of many OpenMP applications that frequently contain computationally-intensive and large data parallel loops. Such OpenMP applications are typically executed on high performance computing (HPC) platforms which are increasingly complex, large, heterogeneous, and exhibit massive and diverse parallelism. The performance of applications executing on HPC platforms can be degraded due to various overheads, such as synchronization, management of parallelism, communication, and load imbalance [4]. Indeed, these overheads cannot be ignored by any effort to improve the performance of applications, such as the loop scheduling schemes [7]. The scheduling of those large and complex OpenMP loops can be a critical factor for the efficient use of those HPC platforms.

The optimal scheduling of parallel applications on parallel computing platforms is NP-hard [16]. No single loop scheduling technique can address all sources of load imbalance to effectively optimize the performance of all parallel applications executing on all types of computing platforms. Indeed, the characteristics of the loop iterations compounded with the characteristics of the underlying computing systems determine, typically during execution, whether a certain scheduling scheme outperforms another. The performance of parallel applications is impacted by system-induced variability (e.g., operating system noise, power capping) and results in additional irregularity that has often been neglected in loop scheduling research, particularly in the context of OpenMP scheduling [17, 30]. Efficient loop scheduling can mitigate those variabilities, if a suitable schedule is available. However, choices for loop scheduling strategies in OpenMP are limited today to static, guided, or dynamic. These three scheduling strategies have been shown in previous work [8, 22] not to offer the best performance possible. Moreover, fault-tolerant and energy-oriented OpenMP loop scheduling strategies require domain-specific knowledge to maintain correctness and energy-efficiency at large-scale, respectively [11, 32], which is currently not exploited by the three standard OpenMP scheduling strategies.

More and novel loop scheduling strategies are needed in OpenMP given complexity of emerging applications and of supercomputer architectures. This is evident by the efforts of compiler developers, open-source and commercial alike, to support additional scheduling schemes. The efforts can be observed in LLVM [1] with the trapezoid self-scheduling [31] strategy, or in the Intel compiler with a static stealing scheme [24]. However, given the great body of work on loop scheduling, in general, standardizing all possible scheduling strategies in OpenMP is infeasible. Therefore, given the many different compilers supporting OpenMP, a standardized way of supporting additional scheduling strategies is mandatory for portability and use in today’s frequently changing HPC landscape.

A more viable approach is to extend the OpenMP standard to allow for user-defined loop scheduling (UDS). Doing so will enable application-specific scheduling as well as a standard-compliant means to customize current loop schedulers. To this end, this work analyzes the principal operations of a loop scheduling scheme using a ‘todo list’ as a representation of the loop iteration space. Based on this modeling we identify four mandatory operations (init, enqueue, dequeue, and finalize). To support all currently available scheduling strategies, additional information may be necessary, which can be obtained through two measurement operations around the loop body. Using these principal components, we propose two complementary UDS specification interfaces for OpenMP, following the distinct styles of C, Fortran, and C++. One proposal supports a more modern programming style, such as that used in C++14 and later. The other proposal takes a classic approach, is suitable for C, Fortran and C++ programs and helps many types of applications to run on various architectures. The aim is that these proposals serve the OpenMP community and compiler developers as a basis for discussion and prototype implementation of UDS.

The core contributions of this work are:

(1) an analysis of existing scheduling strategies and specifications of a minimal function set that is capable of implementing them and

(2) an actual language-specific proposal of how to implement existing and future user-defined scheduling strategies.

The remainder of this paper is structured as follows. First, we provide the background and state of the art in recent loop scheduling strategies in Section 2. We then introduce our proposal for an interface in OpenMP to facilitate user-defined scheduling and its design rationale in Section 3. We present in Section 4.2 the two alternative proposals for the specification of user-defined loop scheduling for OpenMP. Finally, we summarize our experience in Section 5. 111The final publication is available at https://www.springerlink.com.

2 Scheduling Background and State of the Art

Scheduling, as broadly understood, refers to the orchestration of units of work onto units of execution, in space and time. It typically consists of three steps: partitioning, assignment, and load balancing. A computational application is partitioned into units of work to expose the software parallelism. This parallelism is expressed by assigning the units of work (e.g., problem sub-domains) to units of processing (e.g., processes, threads, tasks). The parallel units of processing are subsequently assigned to units of execution (e.g., nodes, processors, cores) to exploit the available hardware parallelism. Load balancing refers to evenly assigning the units of work to units of processing (software load balancing) or to evenly assigning the units of processing to units of execution (hardware load balancing). In load balancing, the transfer policy determines whether a unit of work should be transferred, while the location policy determines where it should be transferred. Based on the location policy, load balancing approaches can be sender-initiated (also called work sharing), receiver-initiated (also referred to as self-scheduling or work stealing), or symmetrically-initiated [23].

Load imbalance is the major performance degradation overhead in computationally-intensive applications [12, 13]. It can result from the uneven assignment of units of computation to units of processing (e.g., threads) or the uneven assignment of units of processing to units of execution. At light and moderate load imbalance, sender-initiated and symmetrically-initiated algorithms outperform receiver-initiated algorithms. Conversely, at high loads, they perform poorly, possibly causing system instability and are outperformed by receiver-initiated algorithms [23]. A load balanced execution refers to the case when all units of execution complete their assigned work at the same time.

In this work, we concentrate on the scheduling and (software) load balancing of parallel OpenMP loops. In this context, we consider computational problems that contain parallel loops expressed using OpenMP worksharing constructs. The iterations of these loops are scheduled and load balanced, respectively, to achieve a load balanced execution.

It is important to note that many scientific, engineering, and industrial applications that use OpenMP contain worksharing loops. Therefore, scheduling of worksharing loops in OpenMP is not overshadowed by the recent advances and developments in OpenMP tasking. Worksharing loops and tasking represent two complementary parallel programming approaches that intersect when each iteration of a worksharing loop creates an OpenMP task to execute the loop body.

The term loop scheduling strategy denotes the technique (or policy) for assigning the loop iterations to threads in a team. A loop scheduler refers to the implementation of a particular loop scheduling strategy, while loop schedule represents the resulting assignment of loop iterations to threads in a team based on the particular scheduling strategy and its corresponding scheduler. In this work, the acronym UDS denotes user-defined loop scheduling. However, unless otherwise noted, the term UDS is also interchangeably used to denote either scheduling, scheduler, or schedule.

There exists a great body of work on loop scheduling and a taxonomy of loop scheduling strategies can be found in recent literature [8]. Loop scheduling strategies can broadly be classified into static and dynamic. The dynamic strategies can further be classified into non-adaptive and adaptive. The static scheduling strategies take the partitioning, assignment, and load balancing decisions before the loop executes, while dynamic scheduling strategies take most of or all these decisions during execution. Moreover, the dynamic adaptive scheduling strategies adapt these decisions as the loop executes based on the application, execution, and system states, to deliver a highly balanced execution.

The OpenMP specification [9] offers three scheduling options for worksharing loops: static, dynamic, and guided. Each can be directly selected as arguments to the OpenMP schedule() clause of a for directive. The first option falls into the static scheduling category, while the other two options belong to the dynamic non-adaptive scheduling category with receiver-initiated load balancing location policy. The loop scheduling strategies can also automatically be selected by the OpenMP runtime system via the auto argument to schedule() or their selection can be deferred to execution time via the runtime argument to schedule().

The use of schedule(static,chunk) employs straightforward parallelization or static block scheduling [25] (STATIC) wherein $N$ loop iterations are divided into $P$ chunks of size $\lceil N/P\rceil$; $P$ being the number of units of processing (e.g., threads). Each chunk of consecutive iterations is assigned to a thread, in a round-robin fashion. This is only suitable for uniformly distributed loop iterations and in the absence of load imbalance. The use of schedule(static,1) implements static cyclic scheduling [25] wherein single iterations are statically assigned consecutively to different threads in a cyclic fashion, i.e., iteration $i$ is assigned to thread $i$ mod $P$. For certain non-uniformly distributed parallel loop iterations, cyclic scheduling produces a more balanced schedule than block scheduling. Both versions achieve high locality with virtually no scheduling overhead, at the expense of poor load balancing if applied to loops with irregular loop iterations or in systems with high variability.

The dynamic version of schedule(static,chunk) that employs dynamic block scheduling is schedule(dynamic,chunk). It differs in that the assignment of chunks to threads is performed during execution. The dynamic counterpart to schedule(static,1) is schedule(dynamic,1) which employs pure self-scheduling (PSS or simply SS), the easiest and most straightforward dynamic loop self-scheduling algorithm [29]. Whenever a thread is idle, it retrieves an iteration from a central work queue (receiver-initiated load balancing). SS achieves good load balancing yet may cause excessive scheduling overhead. The scheduling option schedule(guided) implements guided self-scheduling(GSS) [26], one of the early self-scheduling-based techniques that trades off load imbalance and scheduling overhead.

Further noteworthy dynamic non-adaptive loop scheduling techniques are trapezoid self-scheduling (TSS) [31], factoring2 (FAC2) [15], and weighted factoring2 (WF2) [14]. TSS, FAC2, and WF2 do not require additional information about loop characteristics and the allocated chunk sizes using these techniques decrease during the course of the execution from one work request to another. It is important to note that the FAC2 and WF2 evolved from the probabilistic analyses that conceived FAC [15] and WF [14], respectively, while TSS is a deterministic self-scheduling method. Moreover, WF2 can employ workload balancing information specified by the user, such as the capabilities of a heterogeneous hardware configuration.

TSS, FAC2, WF2, and RAND (random self-scheduling-based method that employs the uniform distribution between a lower and an upper bound to arrive at a randomly calculated chunk size between these bounds) [8] have been implemented in the LaPeSD libGOMP [3] based on the GNU OpenMP library. The LLVM OpenMP runtime [1] also provides an implementation of TSS [31] and static stealing (also referred to as fixed-size chunking [24]). FAC2 has also been recently implemented in the LLVM OpenMP runtime to offer further performance enhancement possibilities at higher loads [22].

This review of existing related efforts shows that there is a large amount of ad-hoc development of loop scheduling strategies and schedulers for OpenMP in various OpenMP runtime libraries (RTLs), yet none of these efforts comply with the OpenMP specification. While these implementations may remain helpful to certain users, applications, and systems, their broad practical usability may be limited, rendering them not useful for supporting the development of novel advanced loop scheduling strategies in OpenMP.

The main challenge is to decouple the loop scheduling strategy from its implementation strategy. Such a decoupling opens the door to a broad range of dynamic adaptive loop scheduling strategies that simply cannot be efficiently implemented in OpenMP RTLs, such as adaptive weighted factoring [6] and adaptive factoring [5] that adapt to changes during execution; strategies that mix static and dynamic scheduling to maintain a balance between data locality and load balance [20, 10]; and fault-tolerant and energy-oriented loop scheduling strategies that require domain-specific knowledge [32, 11].

3 Support for User-defined Scheduling Strategies

Let us consider what is needed to specify an arbitrary scheduling strategy for a parallel loop. The strategy can use a combination of shared data structures, a collection of low-overhead steal work queues, exclusive queues meant for each core, or shared queues from which multiple threads can dequeue tasks each representing a chunk of loop iterations of a parallel loop. To enable the ability to learn from recent execution history, e.g., recent outer iterations, or to make decisions about the scheduling strategy based on information from libraries handling inter-node parallelism, e.g., slack from MPI communication [27], the scheduling strategy needs the ability to pass a call-site specific history-tracking object [19].

To adapt a loop scheduling strategy’s parameters, e.g., chunk size, we provide a mechanism for a UDS to store the history of loop timings or other statistics across loop invocations in an application program, e.g., a simulation time-step of a numerical simulation. Such a mechanism improves productivity for the application programmer. The adjustment of the loop scheduling strategy during execution reduces the need for manual performance tuning and compiler-guided performance tuning, which for certain applications such as those involving sparse matrix vector multiplication is difficult, and for other applications such as a galaxy simulation involving an $N$-body computation, is nearly impossible.

In order to support UDS in OpenMP, we must first understand the principal components of loop scheduling. Figure 1 shows a control flow diagram of the basic loop scheduling code structure. In principle, an OpenMP loop scheduling problem can be represented as a todo list of loop iterations (or chunks of loop iterations), that must somehow be mapped to parallel execution units. To manage such a todo list, and assuming an undefined initial state, three specific operations are required:

1. (a)

a setup operation to generate a known initial state, i.e., the todo list must be created and initialized, 2. (b)

an enqueue operation to place the loop iterations on the todo list, and 3. (c)

a dequeue operation to select the next loop iteration to be executed from the todo list.

As OpenMP requires that the precise iteration space is known before the loop execution starts, the todo list is conceptually completely filled at the beginning of loop execution with all the chunks of loop iterations, and subsequently consumed by iterative dequeue operations by each OpenMP thread. The dequeue operation then implements an arbitrary scheduling strategy or pattern. Constraints, such as sequential ordering or the scheduling pattern are solely an aspect of the dequeue operation as well as any synchronization mechanisms to maintain parallel safety of the used data structures. For both the enqueue and dequeue functions, the master thread can potentially serve a different function than the remaining threads in a loop scheduler. Also, the behavior of the threads needs to be specified either via function pointers or declaratively. Such specification must be done while preserving generality so that novel loop scheduling strategies have the ability to deal with the loop’s iteration space in a controlled manner. As an example, we have shown how dynamic scheduling can be optimized by using a combination of statically scheduled and dynamically scheduled loop iterations [10], where the dynamic iterations still execute in consecutive order on a thread to the extent possible [18].

Good practice also recommends to clean up after performing work, as the OpenMP base languages do not offer automatic garbage collection. Hence, a clean-up, or post scheduling operation is needed.

Analyzing the current state of the art in loop scheduling in Section 2, we identified three categories of strategies:

1. (1)

static loop scheduling: each thread is assigned a fixed workload, 2. (2)

dynamic non-adaptive loop scheduling: each thread requests iterations according to a fixed pattern, and 3. (3)

dynamic adaptive loop scheduling: each thread requests iterations according to a variable pattern, while the performance of work chunks is measured and scheduling pattern is adjusted accordingly.

For loop scheduling strategies of type (1) and type (2), in principle, only the three operations are required. For strategies of type (3), the execution behavior of previous iterations of the loop body is used as input to determine the scheduling strategy parameters, e.g., next chunk size, to use for scheduling chunks of loop iterations of the current loop iteration and/or invocation. To accommodate such scheduling strategies, a mechanism needs to be provided to obtain information during previous loop iterations and/or invocations and a mechanism to store this information.To obtain the information, measurement facilities for the loop body may be required, be it explicit operations, such as ‘begin-loop-body’–‘end-loop-body’ to allow for measurements, or implicit facilities, e.g., as defined by the OpenMP tools interface.

To store information, i.e., a form of execution history that must be preserved across dequeue operations to account for past behavior, UDS must provide a mechanism to store and access the history of loop timings or other statistics across multiple loop iterations and/or invocations in an application program, e.g., across simulation time-steps of a numerical simulation.

With these functions and mechanisms, a user of OpenMP can declare in the code a schedule clause of kind X. In the declaration, the user would specify a function to initialize the scheduler, a function to enqueue chunks onto a shared queue, a function to dequeue chunks of iterations from a queue by a thread, a function for garbage collection (finalize) after loop scheduling is done, and, optionally, begin and end functions for a dynamic adaptive loop scheduling. Then, function X_init() allows a user-defined scheduling to allocate and initialize its data structures that are to be used commonly across parallel loops that use X. The functions X_enqueue() and X_dequeue() determine a loop’s indices that a thread should work on based on the parameter values for the scheduling strategy and the loop. Every thread in the team should call X_dequeue() repeatedly. For adaptive loop scheduling, one needs to have an X_begin() and X_end() function for measurements of the current invocation of a loop used for history used for adapting the parameters of the scheduling strategy used in subsequent iterations and/or invocations of the loop. Finally, a user can optionally define a data structure to store timings of a loop or other data to enable persistence over invocations of an OpenMP parallel loop.

As long as one is allowed to define the four functions (init, enqueue, dequeue, and finalize), together with the begin and end functions for gathering per-loop invocation data and data structure for storing history of the data, one can implement any user-defined loop scheduling through a loop scheduler. Formally, the four functions together with begin and end functions and class declaration and definition for the history object are necessary and sufficient to fully express an arbitrary user-defined loop scheduling strategy.

4 An Interface for User-defined Loop Scheduling

As described in Section 3, only six operations, i.e., init, enqueue, dequeue, finalize, begin-loop-body, and end-loop-body must be defined in order to implement all existing loop scheduling strategies. While not all of those operations must be implemented by a given loop scheduling strategy, it must be possible to implement those operations. An interface for a UDS in OpenMP must enable such definitions from the user program without having to alter the OpenMP runtime library. However, due to a programmer’s desire for brevity, such an interface should avoid verbosity and enable efficient and quick specification of new scheduling strategies.

Due to the restriction and requirements of the OpenMP language on loops, the set of six operations can further be reduced. As the iteration space of loops with OpenMP parallel for must be fixed prior to loop execution, the enqueue function must only be executed prior of the actual loop execution. It, therefore, can be merged with the init operation. The dequeue operation and the begin-loop-body operations are executed, if defined, always back-to-back. Hence, these operations can also be implemented in a single merged operation. The conceptional code transformation (see Fig. 1) in combination with a loop similarly provides a way to merge the end-loop-body operation with the dequeue operation.

This results in only three operations that must be defined by a UDS developer in the context of OpenMP loop scheduling: a start routine implementing the setup and enqueue operation, a get-chunk operation implementing the end-body, dequeue and begin-loop-body operation, and a finish call for the finalize operation.

The concept of a todo list of loop iterations is rather impractical for OpenMP loops, as the iteration space may be large and an explicit enumeration of all iterations is not practical. Thus, the todo list is typically implemented as a set of shared or thread-private loop counters.

For current implementations of OpenMP parallelized loops in Intel, LLVM and GNU Runtime Libraries, we observe a common implementation pattern. Using the three fundamental operations of init, dequeue and finalize, these compilers transform an OpenMP ‘parallel for‘ as follows using the following pattern: a setup operation, a while loop with a dequeue function and a tailing end operation, which implements cleanup of residual stack data (see code at the top of this page). The three OpenMP loop scheduling strategies, i.e. static, guided, and dynamic, are implemented using similar patterns [22].

A UDS specification must allow a loop scheduling implementer to access critical loop parameters and program data:

a) lower bound,

b) upper bound,

c) stride,

d) custom data, e.g. loop history data or NUMA information, and

e) chunk size.

The ‘chunk size’ here is not the chunksize parameter frequently referred to in the OpenMP schedule() clause, but an optimization parameter used to group multiple iterations into a single loop scheduling item.

We currently propose two complementary proposals for an interface for a UDS, enabling a user specification for those three functions. However, the design of these interfaces substantially differs at the OpenMP host language level:

(1) a C++-geared interface using a concept similar to lambdas and

(2) a more classic C/Fortran-geared interface similar to user-defined reductions in OpenMP.

4.1 Lambda-style Specification for UDS

Using a lambda-style syntax, a scheduling implementer can define code to implement the setup, dequeue, and finalize operations.

⬇

#pragma omp parallel for \

schedule(UDS[:chunkSize, [monotonic| non-monotonic]) \

[init(@@INIT_LAMDA@@)] dequeue(@@DEQUEUE_LAMDA@@) \

[finalize(@@FINISH_LAMDA@@)] [uds_data(void*)]

To access the critical loop parameters, we propose compiler-generated getter and setter functions.

⬇

inline unsigned int OMP_UDS_loop_start();

inline unsigned int OMP_UDS_loop_end();

inline unsigned int OMP_UDS_loop_step();

inline unsigned int OMP_UDS_chunksize();

inline unsigned int OMP_UDS_user_ptr();

⬇

void OMP_UDS_loop_chunk_start(int start_iteration);

void OMP_UDS_loop_chunk_end(int end_iteration);

void OMP_UDS_loop_chunk_step(int step_size);

void OMP_UDS_loop_dequeue_done();

To compile a loop scheduled using a UDS, the compiler mixes the lambda code into the respective regions in the loop transformation pattern. The setter and getter functions can furthermore be inlined and their values propagated by constant value propagation, to further reduce and optimize the specific loop code.

As this interface would require a definition for every use of a specific loop scheduling approach, a template-like directive defines reusable schedules without the need to repeat the actual UDS code at every usage.

⬇

#pragma omp declare schedule_template (mystatic) \

[init(@@INIT_LAMDA@@)] dequeue(@@DEQUEUE_LAMDA@@) \

[finalize(@@FINISH_LAMDA@@)] [uds_data(void*)]

⬇

#pragma omp parallel for schedule(UDS,template(mystatic))

for (int i = 0; i < n; i++)

{ ... LOOP BODY ... }

The availability of both UDS templates and localized UDS allows for implementation of libraries supported UDSs, but preserves the ability to either specify localized single use loop scheduling strategies or to overwrite specific elements of an existing UDS template for a specific loop. An example of how the user could implement the above mystatic is provided in Fig. 2 where the left side illustrates a naive implementation of the OpenMP static scheduling clause using lambda-style UDS based on the chunksize specified by the programmer.

4.2 Specifying UDS via declare Directives

The second variant for specifying UDS derives from the existing syntax for a user-defined reduction, or UDR, in OpenMP. Here, the declare schedule clause defines a new named scheduling using user-defined functions with positional arguments:

⬇

#pragma omp declare schedule(mystatic) arguments(2) \

init(my_init(omp_lb, omp_ub, omp_inc, omp_arg0, omp_arg1)) \

next(my_next(omp_lb_chunk, omp_ub_chunk, omp_arg0, omp_arg1)) \

fini(my_fini(omp_arg1))

The arguments sub-clause allows to specify the number of additional arguments beyond the required arguments. The reserved keywords omp_lb, omp_ub, omp_inc, omp_lb_chunk, and omp_ub_chunk serve as markers for the compiler what information about the loop iteration space to pass to the UDS, as the user code expects this information as a function argument. The compiler generates omp_arg0 .. omp_argN as necessary, based on the count in the arguments sub-clause. However, the OpenMP-defined arguments must always be the first arguments, followed by any user-defined arguments. This allows, for example, simpler scheduling strategies to omit unused information. The additional user-provided arguments use the type of the argument at the use-site of the user-defined scheduling, similar to the auto-type in C++. The function implementations must then use the appropriate types and provide the implementation of the scheduling strategy. Please note, that a definition of the next function must return a non-zero value if unprocessed loop chunks remain, and zero if the loop has been completed.

⬇

void mystatic_init(int lb, int ub, int inc, loop_record_t * lr);

int mystatic_next(int * lower, int * upper, loop_record_t * lr);

void mystatic_fini(loop_record_t * lr);

To generate code from such a UDS specification, the compiler employs the standard loop transformation pattern it uses today and replaces the calls to its scheduling function with user-supplied functions of the UDS. The compiler may then match the types defined by the scheduling implementing function definitions to generate error messages, if a type mismatch is detected, or apply inlining to remove the function call.

The following example showcases how a user-defined scheduling strategy would be used and how parameters are passed to the scheduler:

⬇

#pragma omp parallel for schedule(mystatic(&lr))

for (i = 0; i < sz; i++) {

#pragma omp atomic

array[i]++;

}

}

An example of how the user could implement the above schedule mystatic is provided in Fig. 2, where the right side shows a naive implementation of the OpenMP static scheduling clause using declare-style UDS based on the chunksize specified by the programmer.

4.3 Discussion

We consider both proposals sufficient as a UDS specification layer. As OpenMP targets three separate host languages, we must consider the implications of each interface to the host language and use in daily programming work.

The lambda-style interface easily fits into the language canon of C++, where the concept of lambdas already exists and can easily be reused in the context of UDS. Also, the use of getter and setter functions does not present a source of overhead, as existing compiler optimizations, such as inlining and constant-value propagation and folding, will enable removal of all explicit function calls. As some operations, i.e., setup and finalize, are also not required for all implementations of a UDS, this avoids the verbose, potentially empty argument list of positional arguments, required by the second proposal. However, the flexibility and ease of iteration in C++ conflicts with C and Fortran, where lambda constructs are not (yet) available. While the concept of lambdas is likely to be added to Fortran in the future, the specific syntax and semantics are currently not known. At this point, we are also not aware of any efforts to add lambdas to C. The UDR-style specification has, in principle, a precedence-case in the UDR specification in OpenMP. While this approach relies on a more frumpy fixed position syntax style, it remains compatible with all three OpenMP host languages.

A potential solution would allow the use of the lambda-style syntax for C++, and the UDR-style for C and Fortran codes.

Our suggested UDS approach for supporting novel loop scheduling strategies and two alternative interfaces for it have much work related to it, which we mention here to distinguish our idea and its development from the existing work. Work on an OpenMP runtime scheduling [30, 33] system automatically chooses the schedule. The problem with this scheme is that it does not work for all application-architecture pairs: it allows no domain knowledge or architecture knowledge to be incorporated into it, which only a user would know. Methods such as setting the schedule of an OpenMP loop to ‘auto’ are insufficient because the methods do not allow a user to take control of any decision of loop scheduling that the OpenMP RTL makes [21]. The emergence of threaded runtimes such as Argobots [28] and QuickThreads [2] are frameworks containing novel loop scheduling strategies, and they actually argue in favor of a flexible specification of scheduling strategies. In comparison, our work on the UDS specification is the first proposal that works at the OpenMP standard specification level.

5 Conclusion

OpenMP’s loop scheduling choices do not always offer the best performance, and standardization of all existing scheduling strategies is infeasible. In this work, we showed that an OpenMP standard-compliant interface is needed to implement an arbitrary user-defined loop scheduling strategy. We presented two competing standard-compliant UDS interface proposals to support this need. We conceptually compare the two proposed UDS interfaces in terms of feasibility and capabilities regarding the programming languages C, C++, and Fortran that host OpenMP.

The immediate next step is the implementation of the UDS interfaces as a prototype in an open source compiler, such as GNU or LLVM, to explore the performance-related capabilities and benefits of the proposed approaches. As the Intel and LLVM OpenMP RTLs offer schedules choices beyond those in the OpenMP standard, we will work to expose those schedules using either or both UDS proposals and evaluate their practical use for various application-architecture pairs. We welcome and value the feedback from the OpenMP community as we proceed in this direction.

Acknowledgments

We thank Alice Koniges from Maui HPCC for providing us with NERSC’s cluster Cori for experimenting with machine learning applications using OpenMP, which helped us consider a relevant platform for user-defined scheduling. This work is partly funded by the Hessian State Ministry of Higher Education by granting the “Hessian Competence Center for High Performance Computing” and by the Swiss National Science Foundation in the context of the “Multi-level Scheduling in Large Scale High Performance Computers” (MLS) grant, number 169123.