A Fast Causal Profiler for Task Parallel Programs

TL;DR

TASKPROF is a hardware-assisted, fine-grained causal profiler for task parallel programs that helps identify and optimize parallelism bottlenecks efficiently, even before concrete optimizations are known.

Contribution

The paper introduces TASKPROF, a novel causal profiling tool leveraging hardware counters for minimal perturbation and parallel execution, enabling effective bottleneck identification and optimization guidance.

Findings

Successfully identified bottlenecks in 23 applications.

Achieved tenfold increase in parallelism in five applications.

Developers found TASKPROF easy to use for performance analysis.

Abstract

This paper proposes TASKPROF, a profiler that identifies parallelism bottlenecks in task parallel programs. It leverages the structure of a task parallel execution to perform fine-grained attribution of work to various parts of the program. TASKPROF's use of hardware performance counters to perform fine-grained measurements minimizes perturbation. TASKPROF's profile execution runs in parallel using multi-cores. TASKPROF's causal profile enables users to estimate improvements in parallelism when a region of code is optimized even when concrete optimizations are not yet known. We have used TASKPROF to isolate parallelism bottlenecks in twenty three applications that use the Intel Threading Building Blocks library. We have designed parallelization techniques in five applications to in- crease parallelism by an order of magnitude using TASKPROF. Our user study indicates that developers are able to isolate performance bottlenecks with ease using TASKPROF.

Full text

1 Introduction

Task parallelism is an effective approach to write performance portable code Grossman and Anderson [2012]. In this model, the programmer specifies fine-grained tasks and the runtime maps these tasks to processors while automatically balancing the workload using work stealing algorithms. Many task parallelism frameworks have become mainstream (e.g., Intel Threading Building Blocks (TBB) Reinders [2007], Cilk Frigo et al. [1998], Microsoft Task Parallel Library Leijen et al. [2009], Habanero Java Cavé et al. [2011], X10 Charles et al. [2005], and Java Fork/Join tasks Lea [2000]).

A common metric used to quantify the performance of a task parallel program is asymptotic parallelism, which measures the potential speedup when the program is executed on a large number of processors. It is constrained by the longest chain of tasks that must be executed sequentially (also known as the span or the critical work). Hence, asymptotic parallelism is the ratio of the total work and the critical work performed by the program for a given input. A scalable program must have large asymptotic parallelism. A task parallel program can have low asymptotic parallelism due to multiple factors: coarse-grained tasks, limited work performed by the program, and secondary effects of execution such as contention, low locality, and false sharing.

Numerous techniques have been proposed to address various bottlenecks in both multithreaded programs Tallent et al. [2010]; Chen and Stenstrom [2012]; Du Bois et al. [2013a]; Yu and Pradel [2016]; David et al. [2014]; Liu and Mellor-Crummey [2013a]; Liu and Wu [2015]; Liu and Mellor-Crummey [2013b] and task parallel programs Schardl et al. [2015]; He et al. [2010]. These technique range from identifying critical paths Hollingsworth and Miller [1994]; Miller et al. [1990]; Oyama et al. [2000], parallelism Schardl et al. [2015]; He et al. [2010], synchronization bottlenecks Tallent et al. [2010]; Chen and Stenstrom [2012]; Du Bois et al. [2013a]; Yu and Pradel [2016]; David et al. [2014], and other performance pathologies Liu and Mellor-Crummey [2013a]; Liu and Wu [2015]; Liu and Mellor-Crummey [2013b]. Tools for multithreaded programs identify bottlenecks in a specific execution on a specific machine, which does not necessarily provide information about scalability of the program. In contrast, tools that measure asymptotic parallelism in task parallel programs run the program serially Schardl et al. [2015]; He et al. [2010], which is feasible only when the task parallel model provides serial semantics (e.g., Cilk) Frigo et al. [1998]. Although they identify parallelism bottlenecks, they do not provide information on regions of code that matter in improving asymptotic parallelism.

This paper proposes TaskProf, a fast and causal profiler that measures asymptotic parallelism in task parallel programs for a given input. TaskProf’s causal profile allows users to estimate improvements in parallelism when regions of code are optimized even before concrete optimizations for them are known. TaskProf has three main goals: (1) to minimize perturbation (also known as interference Graham et al. [1983]) while accurately computing asymptotic parallelism and critical work for each spawn site (source code location where a task is created), (2) to run the profiler in parallel, and (3) to provide feedback on regions of code that matter in increasing parallelism.

TaskProf computes an accurate parallelism profile by performing a fine-grained attribution of work to various parts of the program using the structure of a task parallel execution. The execution of a task parallel program can be represented as a tree (specifically Dynamic Program Structure Tree (DPST) Raman et al. [2012]), which captures the series-parallel relationships between tasks and can be constructed in parallel. Given a task parallel program, TaskProf constructs the DPST in parallel during program execution and attributes work to leaves of the DPST. To minimize perturbation, TaskProf uses hardware performance counters to measure work performed in regions without any task management constructs, which correspond to leaves in the DPST. TaskProf writes the DPST and the work performed by the leaf nodes of the DPST to a profile data file. The profile execution runs in parallel leveraging multi-cores and the measurement of computation using performance counters is thread-safe.

TaskProf’s post-execution analysis tool uses the data file from the profile run, reconstructs the DPST, and computes asymptotic parallelism and critical work at each spawn site in the program using the properties of the DPST (see Section 3.2). TaskProf maps dynamic execution information to static spawn sites by maintaining information about spawn sites in the DPST. TaskProf’s profile for the sample program in Figure 2 is shown in Figure 3(b).

The spawn sites that perform a large fraction of the critical work in the profile are the parallelism bottlenecks in the program. However, optimizing regions that perform critical work may not increase asymptotic parallelism when the program has multiple regions that perform similar amount of critical work. Designing a parallelization strategy that reduces critical work requires significant effort. Hence, the programmer would like to know if optimizing a region of code increases asymptotic parallelism even before the specific optimization is designed.

TaskProf provides a causal profile that estimates the improvement in asymptotic parallelism when a specific region of code in the program is optimized even before concrete optimizations for them are known. TaskProf’s causal profile is inspired by Coz Curtsinger and Berger [2015] that quantifies the speedup when a selected program fragment is optimized in multithreaded programs by slowing down all code executing concurrently with the fragment. However, Coz cannot be used with task parallel programs as it is not possible to slow down all active tasks.

In contrast, TaskProf is able to generate a causal profile because it builds an accurate performance model of a task parallel execution by performing a fine-grained attribution of work to the nodes of the DPST. To quantify the impact of optimizing a region of code, the programmer annotates the beginning and the end of the region in the program and the anticipated speedup for the region. TaskProf generates a causal profile that shows the increase in parallelism with varying amounts of anticipated speedup for the annotated regions (see Figure 3(c)). To generate a causal profile, TaskProf re-executes the program, generates profile data, and identifies nodes in the DPST that correspond to the annotated regions. Subsequently, TaskProf recomputes the asymptotic parallelism in the program by reducing the critical work of the annotated region of code by the anticipated improvement. TaskProf’s causal profiling enables the programmer to identify improvements in asymptotic parallelism even before the developer actually designs the optimization. Figure 1 illustrates TaskProf’s usage to generate a parallelism profile and a causal profile.

TaskProf prototype is open source and available online Yoga and Nagarakatte [2017]. We have identified parallelism bottlenecks in twenty three Intel TBB applications using the prototype. Using TaskProf’s causal profile, we also designed concrete parallelization techniques for five applications to address the parallelism bottlenecks. Our concrete optimizations increased parallelism in these five applications by an order of magnitude. We conducted a user study involving thirteen undergraduate and graduate students to evaluate the usability of TaskProf. Our results show that the participants quickly diagnosed parallelism bottlenecks using TaskProf.

2 Background

This section provides a quick primer on the tree-based representation of a task parallel execution, which is used by TaskProf to compute parallelism and causal profiles.

Task parallelism. Task parallelism is a structured parallel programming model that simplifies the job of writing performance portable code. In this model, parallel programs are expressed using a small set of expressive yet structured patterns. In contrast to threads, task creation is inexpensive and a task is typically bound to the same thread till completion McCool et al. [2012]. The runtime uses work stealing to map dynamic tasks to runtime threads and balances the workload between threads Frigo et al. [1998]. Task programming models provide specific constructs to create tasks (e.g., spawn keyword in Cilk and spawn function in Intel TBB) and to wait for other tasks to complete (e.g., sync keyword in Cilk and wait_for_all() function in Intel TBB). A sample task parallel program is shown in Figure 2. These models also provide patterns for recursive decomposition of a program (e.g., parallel_for and parallel_reduce) that are built using the basic constructs. Task parallelism is expressive and widely applicable for writing structured parallel programs.

Dynamic program structure tree. The execution of a task parallel program can be represented as a dynamic program structure tree (DPST), which precisely captures the series-parallel relationships between tasks Raman et al. [2012]. Further, the DPST can be constructed in parallel. Since our goal in this paper is to profile the program in parallel, we use the DPST representation of a task parallel execution.

The DPST is a n-ary tree representation of a task parallel execution. There are three kinds of nodes in a DPST: (1) step, (2) async, and (3) finish nodes. The step node represents the sequence of dynamic instructions without any task spawn or sync statements. All computations occur in step nodes. The async node in the DPST represents the creation of a child task by a parent task. The descendants of the newly created task can execute in parallel with the remainder of the parent task. A finish node is created in a DPST when a task spawns a child task and waits for the child (and its descendants) to complete. A finish node is the parent of all async, finish and step nodes directly executed by its children or their descendants.

The DPST, by construction, ensures that two parallel tasks operate on two disjoint sub-trees. DPST’s construction also ensures that all internal nodes are either async or finish nodes. The siblings of a particular node in a DPST are ordered left-to-right to reflect the left-to-right sequencing of computation of their parent task. A path from a node to the root and the left-to-right ordering of siblings in a DPST do not change even when nodes are added to the DPST during execution. The DPST was originally used for data race detection because it allows a race detector to check if two accesses can occur in parallel Yoga et al. [2016]; Raman et al. [2012]. In a DPST, two step nodes $S1$ and $S2$ (assuming $S1$ is to the left of $S2$) can execute in parallel if the least common ancestor of $S1$ and $S2$ in the DPST has an immediate child that is an async node and is also an ancestor of $S1$. In Section 3.2, we will highlight the properties of the DPST that we use to profile programs.

Illustration of the DPST. Figure 3(a) shows the DPST for an execution of the program in Figure 2. The program in Figure 2 will execute the spawn call at line 10 and line 13 once when BASE=n/2, where $n$ is the number of nodes in the tree.

We construct the DPST during program execution as follows. When the main function starts, we add a finish node $F0$ as the root of the DPST to represent the fact that main completes after all the tasks spawned by it have completed. We add a step node $S0$ as the child of the root finish node to capture the initial computations being performed in the main function. On a spawn call at line 24 in Figure 2, we create a finish node $F1$ because it is the first spawn performed by the task. We also add an async node $A0$ as the child of $F1$ to represent the spawning of a task. Any computation by the newly created task will be added as nodes in the sub-tree under the async node $A0$. The operations performed in the continuation of the main task will be added to the right of the async node $A0$ under the finish node $F1$. Hence, the continuation of the main task and newly created task operate on distinct subtrees of the DPST and can update the DPST in parallel.

3 Parallelism Profiler

TaskProf computes the total work, part of the total work done serially (critical work or span), and the asymptotic parallelism at each spawn site in a task parallel program. The key contribution of TaskProf is in fine-grained attribution of work while ensuring that the profile execution is fast, perturbation-free, and accurate. TaskProf accomplishes the goal of fast profile execution by using multi-cores. TaskProf’s profile execution itself runs in parallel and leverages the DPST representation to attribute work to various parts of the program. TaskProf ensures that the profile execution is perturbation-free by using hardware performance counters to obtain information about the computation performed by the step nodes in the DPST. TaskProf also maintains a very small fraction of the DPST in memory during profile execution to further minimize perturbation. TaskProf ensures that the parallelism profile is accurate by capturing spawn sites through compiler instrumentation and by precisely measuring work performed in each step node.

TaskProf computes the parallelism profile in three steps. First, TaskProf provides a modified library for task parallelism that captures information about spawn sites. TaskProf’s compiler instrumentation modifies the calls to the task parallel library in the program to provide information about spawn sites. Second, TaskProf’s profile execution runs in parallel on the multi-core processors, constructs the DPST representation of the execution, and collects fine-grained information about the execution using hardware performance counters. TaskProf writes the profile information to a data file similar to the grof profiler for sequential programs Graham et al. [1982]. Third, TaskProf’s offline analysis tool analyzes the profile data and aggregates the data for each static spawn site. Finally, it computes asymptotic parallelism and critical work for each spawn site.

Static instrumentation and modified libraries. TaskProf provides a modified task parallelism library that constructs the DPST and reads hardware performance counters at the begining and end of each step node. TaskProf uses static instrumentation to instrument the program with calls to the modified task parallel runtime library. In the subsequent offline analysis phase, TaskProf needs to map the dynamic execution information about asymptotic parallelism and critical work to static spawn sites in the program. Hence, TaskProf instruments the spawn sites to capture the line number and the file name of the spawn site. TaskProf’s static instrumentation is currently structured as a rewriter over the abstract syntax tree of the program using the Clang compiler front-end. Our instrumented libraries and compiler instrumentation enable the programmer to use TaskProf without making any changes to the source code.

3.1 Parallel Profile Execution

The goal of the profile execution is to collect fine-grained information about the program to enable a subsequent offline computation of asymptotic parallelism. Typically, programs are profiled with representative production inputs that have long execution times. Hence, a fast profile execution is desirable. Our goal is to ensure that the execution time of the program with and without profiling is similar. Hence, TaskProf profiles in parallel leveraging multi-core processors. To ensure a parallel profile execution, it needs to construct the execution graph in parallel and collect information about the program in a thread-safe manner.

The DPST representation for parallel profile execution. We use the DPST representation to measure work performed by various parts of the program because the DPST can be constructed in parallel. TaskProf constructs the DPST as the program executes the injected static instrumentation and measures the work performed in each step node. The DPST, once constructed, allows TaskProf to determine the dependencies between tasks. This fine-grained attribution of work to the step nodes in the DPST enables TaskProf to compute the parallelism in the program eventually using an offline analysis.

The DPST of the complete task parallel execution has a large number of nodes. Storing the entire DPST in memory during program execution can cause memory overheads and perturb the execution. To address this issue, TaskProf does not maintain the entire DPST in memory. In a library based task parallel programming model, a task is always attached to the same thread. We leverage this property to minimize the footprint of the DPST in memory. TaskProf maintains a small fraction of the nodes that correspond to the tasks currently executing on each thread in memory.

Once a step node of a task completes execution, the work performed in the step node along with the information about its parent node is written to the profile data file and the DPST node can be deallocated. As async nodes do not perform any work, TaskProf writes the information about its parent in the DPST and the spawn site associated with the async node to the profile data file. In contrast to step and async nodes, only parent node information is written to the profile data file for a finish node.

Measuring work with hardware performance counters. To measure the work performed in each step node without performance overhead, TaskProf uses hardware performance counters. Performance counters are model specific registers available that count various events performed by the hardware using precise event-based sampling mechanisms Corporation [2016]. These performance counters can be programmatically accessed. TaskProf can use both the number of dynamic instructions and the number of execution cycles to measure the work done in a step node. Measuring execution cycles allows TaskProf to account for latencies due to secondary effects such as locality, sharing, and long latency instructions. Further, the operations on these counters are thread-safe. TaskProf reads the value of the counter at the beginning and the end of the step node using static instrumentation injected into the program. It calculates the work performed in the step node by computing difference between the two counter values. This fine-grained measurement of work performed in each step node using hardware performance counters along with the construction of the DPST while executing in parallel allows TaskProf to compute a precise, yet fast profile of the program.

The profile data file generated at the end of parallel profile execution contains the work done in each step node. It also contains the information about the parent for each node in the DPST and the spawn site information for each async node. The left-to-right sequencing of nodes is implicitly captured by the order of the nodes in the profile data file.

3.2 Offline Analysis of the Profile Data

TaskProf’s offline analysis reconstructs the DPST using the data from the profile execution and computes the work and critical work (span) for each spawn site in the program. The construction of the DPST from the profile data is fairly straightforward as it contains information about nodes, their parent nodes, and the left-to-right ordering of the nodes. In this section, we describe the computation of work and span for each intermediate node in the DPST given the work performed in the step nodes. We also describe the process of mapping this dynamic information to static spawn sites.

Computing work and critical work for each intermediate node. In the DPST representation, all computation is performed in the step nodes. The step nodes have fine-grained work information from the profile execution. TaskProf needs to compute the total work and the fraction of that work done serially (critical work) for each intermediate node in the DPST. To provide meaningful feedback to the programmer, TaskProf also computes the list of spawn sites that perform critical work and the portion of the critical work performed exclusively by each spawn site.

TaskProf computes the total work and the critical work at each intermediate node by performing a bottom-up traversal of the DPST. The total work performed in the sub-tree at each intermediate node is sum of the work performed by all the step nodes in the sub-tree. In contrast, critical work measures the amount of work that is performed serially. Computing critical work and the set of tasks performing the critical work requires us to leverage the properties of the DPST. Specifically, we leverage the following properties of the DPST to compute the critical work.

• •

The siblings of a node in a DPST are ordered left-to-right reflecting the left-to-right sequencing in the parent task.

• •

Given an intermediate node, all the direct step children of the node execute serially.

• •

All the left step or finish siblings of an async node execute serially with the descendants of the async node.

• •

All the right siblings (and their descendants) of an async node execute in parallel with the descendants of the async node.

Using the above properties of the DPST, the critical work at an intermediate node will be equal to either (1) the serial work done by all the direct step children and the critical work performed by the finish children or (2) the critical work performed by descendants of an async child and the serial work performed by the left step and finish siblings of the specific async child in consideration. Since any intermediate node in the DPST can have multiple async children, TaskProf needs to check if any of the async nodes can contribute to the critical work. For example, consider the intermediate node $F2$ in Figure 3(a) that has two async nodes $A1$ and $A2$. The critical work will be the maximum of (1) the work done by the direct step child $S4$ or (2) the critical work by the async child $A1$ (it does not have any left siblings), or (3) the sum of the critical work by the async child $A2$ and the work done by the step node $S4$, which is the left step sibling of $A2$.

Each async node in the DPST corresponds to a spawn site in the program because async nodes are created when a new task is spawned. Hence, TaskProf computes the list of spawn sites performing critical work by computing the list of async nodes that contribute to the critical work in the sub-tree of the intermediate node.

Algorithm to compute work and critical work. Figure 4 provides the algorithm used by TaskProf to compute the total work, the critical work, and the set of spawn sites contributing to the critical work. The algorithm maintains four quantities with each intermediate node in the DPST: (1) total work performed in the sub-tree under the node (work), (2) the critical work performed in the sub-tree  (c_work), (3) the list of spawn sites that perform the critical work (ss_list), and (4) the part of the critical work that is performed exclusively by the direct children of the node (e_work). The exclusive work of a node is equal to sum total of the work performed by the direct step children and the exclusive work performed by the finish children. We consider the exclusive work performed by a finish node because it is not yet associated with any spawn site. The exclusive work of the current node will eventually be associated with a spawn site. The algorithm does not consider the exclusive work of the async children because it is already associated with a spawn site.

The algorithm traverses each node in the DPST in a bottom up fashion. All step nodes have work information from the profile data. For any intermediate node, the work performed under the sub-tree is the sum of the work performed by all its children (lines 3-4 in Figure 4). For a given intermediate node, TaskProf initially computes the serial work performed in all the step and finish children as the critical work (lines 5-7 in Figure 4). For each async child of the current node, it checks if the serial work done by the async node and its left siblings is greater than the critical work computed until that point (lines 10-15 in Figure 4).

To compute the set of spawn sites performing critical work, each intermediate node also maintains a list of spawn sites and the exclusive work performed by them. The algorithm initially sets the spawn site list for a node to be the union of spawn site lists of its finish children (lines 8-9 in Figure 4). Whenever an async child contributes to the critical work, the spawn site list of the current node is the union of the spawn site list of the async child and the spawn site lists of the finish children that are to the left of the async child (line 17 in Figure 4). When an async child contributes to the critical work, the exclusive work of the current node is equal to sum of the work performed by the left step siblings and the exclusive work performed by the left finish siblings of the async child (line 16 in Figure 4). The algorithm adds the spawn site and the exclusive work performed by the current async node to the node’s spawn site list (lines 20-22 in Figure 4).

After the algorithm completes traversing the entire DPST, the root of the DPST will contain the list of all spawn sites that perform critical work and their individual contribution to the critical work. The root node also contains information about the total work performed by the program, the work that is computed serially by the program, and the exclusive work performed under the entry function of the program (i.e., main).

Aggregating information about a spawn site. A single spawn site may be executed multiple times in a dynamic execution. Hence, TaskProf aggregates information from multiple invocations of the same spawn site. TaskProf computes the aggregate information for each spawn site by performing another bottom-up traversal of the DPST at the end. When it encounters an async node, TaskProf uses a hash table indexed by the spawn site associated with the async node and adds the total work and critical work to the entry. When aggregating this information, TaskProf has to ensure that it does not double count work and critical work when recursive calls are executed. In the presence of recursive calls, a descendant of an async node will have the same spawn site information as the async node. If we naively add the descendant’s work, it leads to double counting as the work and critical work of the current async node already considers the work/critical work of the descendant async node. Hence, when TaskProf encounters an async node in a bottom-up traversal of the DPST, it checks whether the descendants of the async node have the same spawn site information. When a descendant with the same spawn site exists, it subtracts such a descendant’s work and critical work from the entry in the hash table corresponding to the spawn site. Subsequently, TaskProf adds the work and the critical work of the current async node to the hash table.

Profile reported to the user. For each spawn site in the program, TaskProf presents the work, the critical work, the asymptotic parallelism, and the percentage of critical work exclusively done by the spawn site. The asymptotic parallelism of a spawn site is the ratio of the total work and the critical work performed by a spawn site. The spawn sites are ordered by the percentage of critical work exclusively performed by the spawn site. Figure 3(b) illustrates the parallelism profile for the program in Figure 2 that has the DPST shown in Figure 3(a). If a spawn site has low parallelism and performs a significant proportion of the critical work, then optimizing the task spawned by the spawn site may increase the parallelism in the program. This profile information provides a succinct description of the parallelism bottlenecks in the program.

4 Causal Profiling

TaskProf reports the set of spawn sites performing critical work to the user, which highlight the parallelism bottlenecks in the program. A programmer can consider these spawn sites to be initial candidates for optimization to reduce serial computation.

Reducing critical work and the impact on parallelism. Designing a new optimization or a parallelization strategy that reduces the critical work typically requires effort and time. A program may have multiple spawn sites that perform similar amount of critical work. When a set of spawn sites are parallelized to reduce critical work, the resultant execution may have new spawn sites whose critical work is similar to the critical work before the optimization. In such cases, an optimization to a spawn site performing critical work may not improve the asymptotic parallelism in the program. Hence, programmers would benefit from a causal profile of program that identifies the improvement in asymptotic parallelism when certain regions of the code are optimized.

Causal profile with TaskProf. A causal profile provides information on improvements in parallelism when certain parts of the code are parallelized or optimized. TaskProf proposes a technique to generate causal profiles for task parallel programs. The programmer can get an accurate estimate of the improvement in asymptotic parallelism by reducing the serial work in a region of the program using TaskProf’s causal profile. TaskProf provides such an estimate even before the programmer has designed a concrete strategy to parallelize or reduce the serial work in the region of code under consideration. In summary, a causal profile enables the programmer to identify parts of the program that really matter in increasing the asymptotic parallelism. Figure 3(c) provides the causal profile for the program in Figure 2 where the regions under consideration are demarcated by CAUSAL_BEGIN and CAUSAL_END. Next, we describe how TaskProf generates a causal profile leveraging the accurate performance model of a task parallel execution created with the fine-grained attribution of work and the DPST.

Static code annotations. To generate causal profiles, the programmer annotates a static region of code that is considered for parallelization and the expected improvement to the critical work from parallelization. The programmer can provide multiple regions as candidates for optimization. TaskProf generates a causal profile that estimates the improvement in parallelism when all annotated regions are optimized. In addition, TaskProf also generates a causal profile for optimizing each region in isolation. Figure 2 illustrates the regions of code annotated for causal profiling with CAUSAL_BEGIN and CAUSAL_END annotations. If the programmer does not specify the amount of expected improvement for the considered region, TaskProf assumes a default value. If the annotations are nested, the outermost region of code is considered for estimating the benefits.

Profile execution and attribution of work. TaskProf uses these annotations, profiles the program, constructs the DPST to attribute work to various regions, and provides the estimated improvement in asymptotic parallelism from optimizing the annotated regions. During profile execution, TaskProf measures the work performed in the annotated part of the step node and also in parts of the step node that have not been annotated. Hence, each step node can have multiple work measurements corresponding to static regions with and without annotation. TaskProf accomplishes it by reading the performance counter value at the beginning and the end of the each dynamic region. TaskProf maintains a list of work values for each step node and writes it to the profile data file.

Algorithm to generate causal profiles. The algorithm to compute the causal profile is similar to the work and span algorithm in Figure 4. It takes the DPST as input and a list of anticipated improvements for the annotated regions. The algorithm outputs a causal profile that computes the improvement in asymptotic parallelism of the whole program for the specified improvements of the annotated regions. The causal profile algorithm performs a bottom up traversal of the DPST similar to the work and span algorithm in Figure 4. However, the causal profiling algorithm does not track spawn sites and computes the whole program’s work and critical work. The key difference with the causal profiling algorithm is the manner in which it handles the work done by the step nodes, which have regions corresponding to user annotations. Specifically, TaskProf maintains a list of annotated and non-annotated regions executed with each step node and the amount of work performed in each region. To estimate the effect of optimizing/parallelizing the annotated region, we reduce the critical work contribution of the annotated region by the user-specified optimization factor while keeping the total work performed by the regions unchanged. The output of the causal profiling algorithm is a list that provides the asymptotic parallelism for each anticipated improvement factor for the regions under consideration.

**Illustration. ** After analyzing the parallelism profile in Figure 3(b) for the program in Figure 2, the programmer has identified two regions of code (lines 4-6 and lines 20-22 in Figure 2) for optimization. The regions are annotated with CAUSAL_BEGIN and CAUSAL_END annotations to demarcate the beginning and the end. During execution, the region at lines 20-22 is executed once and is represented by step node $S0$ in Figure 3(a). In contrast, the region at lines 4-6 is executed twice and is represented by step nodes $S5$ and $S6$ in Figure 3(a). In this example, the entire step node corresponds to the annotated region. In general, a step node may have multiple annotated and non-annotated regions. To generate a causal profile, the critical work performed by nodes $S0$, $S5$, and $S6$ are decreased by 2$\times$, 4$\times$, and 8$\times$ and its impact on whole program parallelism is computed. Figure 3(c) provides the causal profile with the annotated regions, which reports that the asymptotic parallelism in the program increases when those two regions are optimized.

5 Experimental Evaluation

This section describes our prototype, our experimental setup, and an experimental evaluation to answer the following questions: (1) Is TaskProf effective in identifying parallelism bottlenecks? (2) Is TaskProf’s parallel profile execution faster than serial profilers? (3) Is TaskProf effective in minimizing perturbation in the profile execution? (4) Is TaskProf usable by programmers?

Prototype. We have built a TaskProf prototype to profile task parallel programs using the Intel Threading Building Blocks(TBB) library Reinders [2007]. The prototype provides a TBB library that has been modified to construct the DPST, measure work done in step nodes using hardware performance counters, and track file name and line information at each spawn site. The prototype also handles algorithms for geometric decomposition such as parallel_for and parallel_reduce. The prototype also includes a Clang compiler pass that automatically adds line number and file name information to the TBB library calls, which enables the programmer to use the modified library without making any source code changes. Hence, the modified TBB library can be linked to any TBB program. Our prototype adds approximately 2000 lines of code to the Intel TBB library to perform various profiling operations. The TaskProf prototype is open source Yoga and Nagarakatte [2017].

Applications used for evaluation. We evaluated TaskProf using a collection of twenty three TBB applications, which include fifteen applications from the problem based benchmark suite (PBBS) Shun et al. [2012], all five TBB applications from the Parsec suite Bienia et al. [2008], and three TBB applications from the structured parallel programming book McCool et al. [2012]. The PBBS applications are designed to compare different parallel programming methodologies in terms of performance and code. We conducted all experiments on a 2.1GHz 16-core Intel x86-64 Xeon server with 64 GB of memory running 64-bit Ubuntu 14.04.3. We measured wall clock execution time by running each application five times and use the mean of the five executions to report performance. We use the perf events module in Linux to programmatically access hardware performance counters.

RQ1: Is TaskProf effective in identifying parallelism bottlenecks? We used TaskProf to identify parallelism bottlenecks in all the 23 applications. Table 1 provides details on applications used, their speedup on a 16-core machine compared to serial execution, the asymptotic parallelism reported by TaskProf, the number of regions that we identified using TaskProf to increase asymptotic parallelism, and the resultant asymptotic parallelism from causal profiling when the critical work in the identified regions is decreased by $100\times$. Typically, asymptotic parallellism of a program should be at least 10$\times$ or more than the anticipated speedup on a machine to account for scheduling overheads Frigo et al. [1998]; Reinders [2007]; Schardl et al. [2015].

TaskProf’s profile shows that some applications in Table 1 have reasonable asymptotic parallelism, which accounts for a reasonable speedup on a 16-core machine. For example, fluidanimate application has an asymptotic parallelism of 66.09 which is the maximum possible speedup when the program is executed on a large number of machines. The fluidanimate application exhibits a speedup of 9.39$\times$ compared to a serial execution when the program was executed on a 16-core machine.

Table 1 also shows that we were able to identify a small number of code regions which when optimized provide a significant increase in asymptotic parallelism. TaskProf’s profile information on spawn sites performing critical work and the causal profiling strategy was instrumental in identifying the specific regions of code as candidates for increasing asymptotic parallelism. The application maxMatching already had a large amount of asymptotic parallelism and we could not find any region that increases parallelism. We were not aware of these parallelism bottlenecks even though these applications have been widely used, which emphasizes the need for TaskProf.

In summary, TaskProf enabled us to identify a set of code regions that can increase asymptotic parallelism significantly in almost all our applications. Once we identified code regions that can increase asymptotic parallelism, we designed concrete parallelization strategies to reduce the critical work for five applications, which increased the asymptotic parallelism and the speedup of the program. We describe them below.

Improving the speedup of the MinSpanningForest application. This PBBS application computes the minimum spanning forest of the input undirected graph. The program has a speedup of 3.47$\times$ over serial execution on a 16-core machine. The parallelism profile generated by TaskProf is shown in Figure 5(I)(a), which reports that the parallelism in the program (main function at MSTTime.C:77) is 7.99. The main function performs 85% of the serial work in the program. We identified two regions of code using annotations for causal profiling in the main function. Figure 5(I)(b) presents the causal profile generated by TaskProf, which shows the increase in asymptotic parallelism in the program on potentially optimizing these two regions. On further investigation of the code regions, we realized that annotated regions were performing a serial sort. We replaced them with a parallel sort function, which increased the asymptotic parallelism to 33.34 from 7.99. Figure 5(I)(c) reports the profile after our parallel sort optimization. The speedup of the program increased from 3.49$\times$ to 6.37$\times$.

Improving the speedup of the Convex Hull application. This PBBS application computes the convex hull of a set of points using a divide and conquer approach Barber et al. [1996]. TaskProf’s profile shown in Figure 5(II)(a) reveals that the program has an asymptotic parallelism of 1.28 for the whole program. As expected, it did not exhibit any speedup. Figure 5(II)(a) shows that 80% of the critical work is performed by the spawn site at hullTime.C:55. We annotated two regions corresponding to that spawn site, which performed sequential read and write operations of the input and output files respectively. TaskProf’s causal profile showed that it would increase the parallelism to 6.85. Subsequently, we annotated two additional regions of code corresponding to the spawn site performing the next highest critical work (hull.C:209) in Figure 5(II)(a). The causal profile shown in Figure 5(II)(b) shows that asymptotic parallelism increases significantly when all the four regions are optimized. We parallelized a loop at spawn site hull.C:209 using parallel_for and parallelized I/O at spawn site hullTime.C:55. These optimizations increased the parallelism to 50.84 (see Figure 5(II)(c)) and the speedup of the whole program increased from 1.3$\times$ to 8.14$\times$.

Improving the speedup of Delaunay Triangulation. This PBBS application produces a triangulation given a set of points such that no point lies in the circumcircle of the triangle. The program has an asymptotic parallelism of 1.47 (see Figure 5(III)(a)) for the entire program and exhibits little speedup. The spawn site at delTime.C:55 performs 99% of the critical work. When we looked at the source code, we found that the program is structured as a collection of parallel_for constructs interspersed by serial code. We annotated five regions of code between the invocations of parallel_for. The causal profile in Figure 5(III)(b)) shows that the asymptotic parallelism increases significantly by optimizing the annotated regions. We parallelized the annotated regions, which had serial for loops, using parallel_for while ensuring they operate on independent data. The profile for the resultant program is shown in Figure 5(III)(c). The parallelism increased to 53.11 and the speedup increased from 1.23$\times$ to 5.82$\times$.

Improving the speedup of Delaunay Refinement. This PBBS application takes a set of triangles that form a delaunay triangulation and produces a new triangulation such that no triangle has an angle less than a threshold value. TaskProf’s profile for this program reports an asymptotic parallelism of 5.5 (see Figure 5(IV)(a)) and it had a speedup of 2.93$\times$. Similar to delaunay triangulation, this program also had a set of serial code fragments in-between parallel_for calls. We identified seven regions of such serial code and annotated them. TaskProf’s causal profile shown in Figure 5(IV)(b) indicates that optimizing all these seven regions can increase asymptotic parallelism. We parallelized the serial for loops in these seven regions using parallel_for, which increased the asymptotic parallelism to 48.08 (see Figure 5(IV)(c)) and the speedup increased from 2.93$\times$ to 6.42$\times$.

Improving the speedup of Blackscholes. This application from the PARSEC suite Bienia et al. [2008] computes the price of a portfolio of options using partial differential equations. It has low asymptotic parallelism for the entire program (see Figure 5(V)(a)). This program has a single parallel_for that has reasonable parallelism of 51.2. However, the spawn site at bscholes.c:323 is performing 99% of the program critical work. Our examination of the code revealed that it was reading and writing serially. We split the input and output into multiple files and parallelized the input/output operations which increased the parallelism to 40.03 and the speedup increased from 1.09$\times$ to 7.7$\times$.

In summary, TaskProf enabled us to quantify asymptotic parallelism in the program and its causal profiling strategy enabled us to identify specific regions of code that can increase parallelism.

RQ2: Is TaskProf’s parallel profile execution faster than serial profile execution? TaskProf’s profile execution executes in parallel compared to prior profilers Schardl et al. [2015], which execute serially. To quantify the benefits of parallel profile execution, we designed a serial version of TaskProf by pinning the execution of the program to a single core. This is an approximation of serial profiling as TBB programs do not have serial semantics. Figure 6 reports the speedup of a parallel TaskProf profile execution compared to a serial profile execution. On average, TaskProf’s parallel profile execution is 4.32$\times$ faster than serial profile execution. The speedup from a parallel profile execution is proportional to the amount of parallelism in the application.

RQ3: Is TaskProf effective in minimizing perturbation in the profile execution? TaskProf uses hardware performance counters to perform fine-grain attribution of work and to minimize perturbation. The average performance overhead of TaskProf’s profile execution compared to the parallel execution of the program without any profiling instrumentation is 56%. A major fraction of this performance overhead is attributed to system calls to read hardware performance counters. TaskProf’s profile execution is an order of magnitude faster than instrumenting each dynamic instruction through compiler instrumentation, which exhibited overheads of 20$\times$-100$\times$ for the applications in Table 1. Hence, TaskProf minimizes perturbation even with fine-grained attribution of work.

RQ4: Is TaskProf usable by programmers? We conducted a user study to evaluate the usability of TaskProf. The user study had thirteen participants: twelve graduate students and one senior undergraduate student. Among them, two students had 4+ years of experience in parallel programming, five students had some prior experience, four students had passing knowledge, and two students had no prior experience with parallel programming. The total duration of the user study was four hours. To ensure that every student had some knowledge in parallel programming, we provided a 2-hour tutorial on task parallelism, and on writing and debugging task parallel programs using Intel TBB. We gave multiple examples to demonstrate parallelism bottlenecks.

After the tutorial, the participants were given a total of four applications and were asked to identify parallelism bottlenecks without using TaskProf in a one hour time period. Three applications — minSpanForest, convexHull, and blackscholes — from Table 1 and a treesum application similar to the example in Figure 2. We chose these applications as it had varying levels of difficulty in diagnosing parallelism bottlenecks. We asked the participants to identify the static region of code causing the bottleneck and record the time they spent to analyze each program. They were not required to design any optimization. Some participants used gprof and others used fine-grained wall clock based timing for assistance. At the end of the time period, twelve of them did not correctly identify parallelism bottlenecks in any of the four applications. One participant, who had 4+ years of experience in parallel programming, identified the bottleneck in one (minSpanForest) out of the four applications. Some participants were misled by the gprof profile.

Subsequently after the first part, we gave a brief tutorial of TaskProf on a simple example program. The participants were then asked to identify bottlenecks in the four applications using TaskProf within an hour. Using TaskProf, seven participants found the parallelism bottleneck in all the four applications, one participant found the bottleneck in three of them, four participant found the bottleneck in two of them, and one participant did not find the bottleneck in any application. Among the participants who identified at least one bottleneck for any application, it took them 12 minutes on average per application to identify the bottleneck using TaskProf. The participants indicated that once they became familiar with the tool by identifying a bottleneck in one application, subsequent tasks were repetitive. In summary, our user study suggests that TaskProf can enable both expert and relatively inexperienced programmers identify parallelism bottlenecks quickly.

6 Related Work

There is a large body of work to identify parallelism bottlenecks. These include techniques to address load imbalances Tallent et al. [2010]; DeRose et al. [2007]; Oh et al. [2011]; Kambadur et al. [2014], scalability bottlenecks Tallent and Mellor-Crummey [2009]; Schardl et al. [2015]; Liu and Wu [2015], visualizing bottlenecks Du Bois et al. [2013b, a]; Eyerman et al. [2012]; Knüpfer et al. [2008], synchronization bottlenecks Chen and Stenstrom [2012]; David et al. [2014]; Yu and Pradel [2016], and data locality bottlenecks Liu and Mellor-Crummey [2011, 2013b]; Acar et al. [2000]. Data locality and synchronization bottlenecks increase serial work. Hence, TaskProf will report asymptotic parallelism in their presence. In contrast to prior proposals, TaskProf also estimates the improvement in parallelism with causal profiling. Next, we focus on the closest related work.

Profiling tools for task parallel programs. Profiling tools such as HPCToolkit Adhianto et al. [2010], and Intel VTune Amplifier Corporation [2017] can analyze a program’s performance on various parameters using hardware performance counters. HPCToolKit also has metrics to quantify idleness and the scheduling overhead Tallent and Mellor-Crummey [2009] in Cilk programs that is specific to a machine. They do not compute the asymptotic parallelism in the program. They also do not identify code that matters with respect to asymptotic parallelism. CilkView He et al. [2010] computes the whole program asymptotic parallelism. CilkProf Schardl et al. [2015] computes asymptotic parallelism per spawn site using an online algorithm. However, these profilers execute the program serially, which is only possible with Cilk programs with C-elision Frigo et al. [1998]. Many task parallelism frameworks including Intel TBB do not have serial semantics, which limits their use. Further, executing the profiler serially can cause high overheads. Unlike TaskProf, they also cannot estimate the benefits of optimizing specific regions of code.

Performance estimation tools. An early profiling technique proposed Slack Hollingsworth and Miller [1994], which is a metric that estimates the improvement in execution time through critical path optimizations for a specific machine model. Kremlin Garcia et al. [2011] identifies regions of code that can be parallelized in serial programs by tracking loops and identifying dependencies between iterations. Kismet Jeon et al. [2011] builds on Kremlin to estimate speedups for the specific machine on which the serial program is executed. These techniques are tied to a specific machine and cannot estimate asymptotic parallelism improvements.

Our work is inspired by Coz Curtsinger and Berger [2015], a causal profiler for multithreaded programs that automatically identifies optimization opportunities and quantifies their impact on a metric of interest, such as latency or throughput. It runs periodic experiments at runtime that virtually speed up a single randomly selected program fragment. Virtual speedups produce the same effect as real speedups by uniformly slowing down code executing concurrently with the fragment, causing the fragment to run relatively faster. In a task parallel context, it is not possible to slow down all active tasks. Further, slowing down threads does not measure the impact of the region as work stealing dynamically balances the load. Further, Coz’s virtual speedups are specific to a particular machine. TaskProf, though similar in spirit, addresses the above challenges and proposes a causal profiler that leverages the dynamic execution structure and estimates improvements in asymptotic parallelism. Hence, TaskProf’s profile is not specific to a single machine and enables the development of performance portable code.

7 Conclusion

TaskProf identifies parallelism bottlenecks by performing a low-overhead, yet fine-grained attribution of work to various parts of the program using the dynamic execution structure of a task parallel execution. TaskProf reports asymptotic parallelism and serial work performed at each spawn site. TaskProf’s causal profile estimates the improvements in parallelism when regions of code annotated by the programmer are optimized. We have identified bottlenecks and improved the speedup in numerous Intel TBB applications. Our user study shows that developers can quickly identify parallelism bottlenecks using TaskProf.

{acks}

We thank the anonymous reviewers for their feedback. This paper is based on work supported in part by NSF CAREER Award CCF–1453086, a sub-contract of NSF Award CNS–1116682, and a NSF Award CNS–1441724.