COMET: A Framework for Modeling Compound Operation Dataflows with Explicit Collectives

TL;DR

COMET is a framework that models and optimizes dataflows involving compound operations with explicit collective communication, improving performance and energy efficiency for modern machine learning accelerators.

Contribution

It introduces a novel explicit modeling approach for collective communication in compound operations, enabling better optimization of dataflows in ML accelerators.

Findings

Achieves up to 1.42× speedup for GEMM-Softmax

Achieves up to 3.46× speedup for GEMM-LayerNorm

Achieves up to 1.82× speedup for self-attention

Abstract

Modern machine learning accelerators are designed to efficiently execute deep neural networks (DNNs) by optimizing data movement, memory hierarchy, and compute throughput. However, emerging DNN models such as large language models, state space models increasingly rely on compound operations-structured compositions of multiple basic operations-which introduce new challenges for dataflow optimization and minimizing off-chip memory traffic. Moreover, as model size continues to grow, deployment across spatially distributed compute clusters becomes essential, requiring frequent and complex collective communication. Existing dataflow optimization frameworks and performance models either focus on single operations or lack explicit modeling of collective communication cost, limiting their applicability to modern workloads. To address these limitations, we propose, a framework for modeling and optimizing dataflow for compound operations on machine learning accelerators. COMET introduces a novel representation that explicitly models collective communication across spatial clusters, along with latency and energy cost models that account for both GEMM and non-GEMM operation level dependencies within compound operations. We demonstrate COMET's capabilities to analyze and optimize dataflows for compound operations such as GEMM--Softmax, GEMM--LayerNorm, and self-attention, across both edge and cloud accelerator configurations. Our collective-aware modeling enables exploration of a broader mapping space, leading to improved performance and energy efficiency. Specifically, our optimized dataflows achieve up to 1.42$\times$ speedup for GEMM-Softmax, 3.46$\times$ for GEMM-LayerNorm and 1.82$\times$ for self-attention compared to unfused baselines.

Tables

Table 1. TABLE I: GEMM shapes for edge architecture.

GEMM ID	GEMM Shape (M-N-K)
GEMM1	1-1024-64
GEMM2	1-4096-128
GEMM3	256-1024-128
GEMM4	4-1024-128
GEMM5	512-1024-128
GEMM6	512-1024-64

Table 2. TABLE II: GEMM shapes for cloud architecture.

GEMM ID	GEMM Shape (M-N-K)
GEMM7	1-16384-128
GEMM8	1-2048-64
GEMM9	256-4096-128
GEMM10	4-8192-128
GEMM11	512-2048-64
GEMM12	512-4096-128

Table 3. TABLE III: Attention operations for edge architecture. The batch size and number of heads is set to 1.

Attention ID	Q ($M\times K$)	K⊤ ($K\times N$)	V ($N\times L$)
Attn1	$1024\times 256$	$256\times 1024$	$1024\times 256$
Attn2	$1\times 128$	$128\times 1024$	$1024\times 128$
Attn3	$1\times 256$	$256\times 2048$	$2048\times 256$
Attn4	$1\times 256$	$256\times 512$	$512\times 256$
Attn5	$256\times 128$	$128\times 256$	$256\times 128$
Attn6	$512\times 128$	$128\times 256$	$256\times 128$

Table 4. TABLE IV: Attention operations for cloud architecture. The batch size and number of heads is set to 1.

Attention ID	Q ($M\times K$)	K⊤ ($K\times N$)	V ($N\times L$)
Attn7	$1024\times 512$	$512\times 1024$	$1024\times 512$
Attn8	$1\times 128$	$128\times 16384$	$16384\times 128$
Attn9	$1\times 512$	$512\times 4096$	$4096\times 512$
Attn10	$1\times 128$	$128\times 8192$	$8192\times 128$
Attn11	$2048\times 256$	$256\times 2048$	$2048\times 256$
Attn12	$256\times 512$	$512\times 256$	$256\times 512$

Table 5. TABLE V: Hardware Configuration. Channel width is defined as number of links.

Parameter	Edge	Cloud
DRAM	1GB (25 GB/s)	4GB (50 GB/s)
Cluster (Mesh)	2$\times$2	4$\times$4
Core (Mesh)	2$\times$2	4$\times$4
Global Buffer (GB)	2 MB (2 TB/s)	8 MB (4 TB/s)
Input and Weight Buffer (IB/WB)	32 KB (4 TB/s)	32 KB (4 TB/s)
Output Buffer (OB)	128 KB (4 TB/s)	128 KB (4 TB/s)
Channel Width (W)	256	2048
$t_{router}$	5 ns	5 ns
$t_{enq}$	2 ns	2 ns
Channel Bandwidth	64 GB/s	512 GB/s

Equations

$$\mathrm{MemLat}=\frac{\text{DV}}{\text{BW}}$$

$$\mathrm{Lat}(T_n)=N\times MW+\mathrm{CS}+\mathrm{OS}$$

$$\mathrm{NoCLat}=t_{\text{router}}\times \text{hops}+t_{\text{enq}}\times \frac{DV}{W}$$

$$\mathrm{Lat}(C{O}_n)=\mathrm{MemLat}+\mathrm{NoCLat}$$

$$\mathrm{Lat}(T_n^0)=\{\begin{array}{ll}\mathrm{Lat}(T_{n-1}^0)+\mathrm{Lat}(T_{n-1}^1),& \text{if sequential}\\ \max (\mathrm{Lat}(T_{n-1}^0),& \\ \mathrm{Lat}(T_{n-1}^1))+\mathrm{conflictStall},& \text{otherwise}\end{array}$$

$$\text{conflictStall}=\{\begin{array}{ll}0,& \text{if }\text{conflictTime}<0\\ \text{conflictTime},& \text{otherwise}\end{array}$$

$$\begin{array}{rl}\text{conflictTime}=& \mathrm{MemLat}(T_{n-1}^0)+\mathrm{MemLat}(T_{n-1}^1)\\ & -\mathrm{Max}(\mathrm{Lat}(T_{n-1}^0),\mathrm{Lat}(T_{n-1}^1))\end{array}$$

Full text

To address these limitations, we propose, a framework for modeling and optimizing dataflow for compound operations on machine learning accelerators. COMET introduces a novel representation that explicitly models collective communication across spatial clusters, along with latency and energy cost models that account for both GEMM and non-GEMM operation level dependencies within compound operations. We demonstrate COMET ’s capabilities to analyze and optimize dataflows for compound operations such as GEMM–Softmax, GEMM–LayerNorm, and self-attention, across both edge and cloud accelerator configurations. Our collective-aware modeling enables exploration of a broader mapping space, leading to improved performance and energy efficiency. Specifically, our optimized dataflows achieve up to 1.42$\times$ speedup for GEMM-Softmax, 3.46$\times$ for GEMM-LayerNorm and 1.82$\times$ for self-attention compared to unfused baselines.

I Introduction

The rapid evolution of machine learning (ML) accelerators has enabled efficient execution of deep neural networks (DNNs) by optimizing compute throughput, data movement and memory hierarchy [28]. Modern accelerators adopt specialized dataflows—strategies for optimizing the scheduling and moving data across the memory hierarchy—to maximize hardware utilization. For instance, Google’s Tensor Processing Unit (TPU) adopts a weight-stationary dataflow [12], while accelerators like Eyeriss use a row-stationary dataflow [5]. However, as DNN models grow in complexity, traditional dataflow strategies face new challenges in optimizing performance.

Emerging models such as large language models (LLMs), state space models (SSMs), and their hybrids have become central to tasks like sequence modeling [26, 10], reasoning, and generative tasks [22]. These models increasingly rely on compound operations—modular blocks that integrate multiple elementary operations such as matrix multiplication, normalization and element-wise transformations [27]. Although these compound operations improve model accuracy and expressiveness [3], they introduce new challenges for dataflow optimization and hardware design [16]. First, the discrepancy between the speed of on-chip compute units and off-chip memory creates a significant performance bottleneck [8]. Second, as model sizes continue to grow, computation must be distributed across multiple compute cores on a single accelerator or even across multiple accelerators [2]. This distributed execution requires expensive collective communication operations, such as All-Reduce and All-Gather [30], to synchronize intermediate results across spatial compute cores. Addressing these challenges requires innovation across both hardware and software stacks. At the circuit level, compute-in-memory techniques have been explored to reduce memory access overheads [24]. At the software level, several works have focused on developing efficient fusion dataflows—execution strategies that combine multiple operators while staging intermediate tensors on-chip [6, 1]. For instance, FlashAttention [6] introduces a fused dataflow for executing attention layers on GPUs. Fused-Layer [1] proposes a tile-stationary dataflow that fuses convolution operations by staging intermediate activations in on-chip memory. Similarly, FLAT [14] designs a fusion dataflow specifically optimized for execution of attention layers on ML accelerators. On the other hand, several works have looked at optimizing collective communication by computation-communication kernel fusion [21, 11] and algorithm design for common collective communications [30].

Designing efficient dataflows for compound operations requires capturing the data-level dependencies between their constituent sub-operations. Prior works have developed cost models for accelerating single operators [20, 17], estimating compute costs, memory transactions, and network-on-chop (NoC) communication overheads. For compound operators, TileFlow [32] proposed a tree-based analysis to estimate the performance of fusion dataflows, while LoopTree [9] explored the tradeoff between retention and recomputation of intermediate tensors during fusion. However, these frameworks inherit limitations from single-operator cost models such as Timeloop [20], and do not explicitly model the cost of collective operations, which becomes critical as compound operations scale and are distributed across spatial clusters. Additionally, they do not account for data staging inefficiencies and assume a single type of compute unit in the hardware, limiting their applicability to modern ML accelerators with specialized GEMM and non-GEMM compute units [7, 15].

To address these limitations, we propose COMET, a framework for modeling and optimizing dataflows for compound operations deployed on ML accelerators with multiple compute cores. COMET captures the interactions between collective communication and operation-level fusion, enabling more enhanced performance modeling and optimization for distributed DNN inference. The design space explored by COMET is illustrated in Fig.1. It spans different types of loop tiling, ordering, and spatial unrolling optimizations (Fig.1(a)); different types of collective operations required for the correct execution of compound operations (Fig.1(b)); the fusion levels of elementary operations within compound operations (Fig.1(c)); and finally, the scheduling strategies across operations simultaneously executing on different compute units (Fig. 1(d)). To the best of our knowledge, COMET is the first framework to explicitly model the cost of collective communication alongside the dataflow optimization of compound operations for ML workloads. Our contributions are summarized as follows:

• •

We propose a representation to explicitly model the cost of collective operations within the dataflow of compound operations (Section IV-A).

• •

We develop a cost model that considers data staging inefficiencies, combined with explicit collective costs, enabling more accurate modeling of compound operation performance (Section IV-B).

• •

We implement the proposed representation and cost model into a framework called COMET, supporting dataflow modeling and mapping exploration for compound operations (Section IV).

• •

We demonstrate the efficacy of COMET through case studies on compound operations commonly found in LLMs, including the GEMM-Softmax and GEMM-LayerNorm blocks (Section V).

II Background

Compound Operations: In modern DNN workloads [26, 10], many layers are structured as compound operations. A compound operation is a collection of multiple primitive operations grouped together for modularity, efficiency, and expressiveness [27]. A common example is the self-attention mechanism, which integrates GEMM, softmax normalization and elemnt-wise scaling into a single high-level operator. Similarly, normalization layers combine non-GEMM operations such as statistics computation and affine transforms [15]. Fig. 2(a) illustrates the self-attention operation from FlashAttention [6], decomposed into a sequence of block-wise GEMM and non-GEMM operations, while also highlighting the reuse of intermediate tensors between successive computation steps.

DNN Accelerators: DNN accelerators are specialized architectures designed to execute DNN workloads with high energy efficiency and throughput. A typical architecture of such an accelerator is shown in Fig. 2(b). These accelerators consist of spatially distributed compute clusters and cores to exploit the inherent parallelism in DNN applications. To reduce energy and latency overhead of DRAM accesses, each cluster consists of a global buffer (GB), which is fed by data from DRAM. The spatial clusters are interconnected through a network-on-chip (NoC) which helps to communicate data across clusters. Each global buffer provides data to input, weight and output buffers (IB,WB,OB) inside the cores, which in turn fill or drain data from GEMM and SIMD compute units. The cores themselves are also connected via a NoC, enabling communication at the lower level of the memory hierarchy. The GEMM unit within each core can be implemented using various architectures, including systolic array [12] or compute-in-memory array [19].

Dataflow and Mapping: Dataflow defines the execution order of the DNN operations on DNN accelerators [20]. Dataflows are often represented in a loop-nest form [5], as shown in Fig. 2(c). In such representations, two types of loops are distinguished: temporal loops (Tp_for), which describe how data moves over time, and spatial loops (Sp_for), which represent how data is unrolled and distributed across hardware units. Different dataflows can be achieved by changing the loop tiling factor, loop order and spatial unrolling [20, 17]. In addition to optimizing these loop transformations, fusion dataflows have emerged as a critical technique for efficient execution. Fusion aims to combine multiple primitive operations into a single fused kernel by staging intermediate tensors on-chip, thereby reducing off-chip memory traffic and improving data locality [14, 1, 6]. For example, fusion strategies are commonly applied in attention layers, where matrix multiplication, softmax, and element-wise scaling operations are fused to minimize data movement. Mapping is an instance of dataflow with a valid tiling factor and ordering resulting in a complete description of the dataflow needed to map the DNN operation onto an accelerator.

Collective Operations: Collective communication operations enable coordinated data exchange among multiple compute clusters or accelerators during the execution of distributed DNN workloads. Common collectives include All-Reduce, All-Gather, Reduce-Scatter, and Broadcast [31]. These operations are critical for maintaining data consistency and enabling intermediate result sharing across spatial compute cores.

III Related Works

Prior works have explored modeling the performance of compound operations in DNNs [1, 14, 32, 9]. Fused-Layer [1] proposes a framework to optimize the execution of convolutional neural networks (CNNs) by fusing multiple adjacent layers to minimize off-chip memory accesses. FLAT [14] develops a fusion dataflow specifically for attention layers in Transformer models, proposing to retain a block of data in on-chip memory to efficiently support the softmax operator. TileFlow [32] presents a tree-based analysis to model and search for efficient dataflows for compound operations. LoopTree [9] explores the tradeoff between recomputation and retention of intermediate tensors to further reduce on-chip buffer requirements when executing compound operations.

While these prior works address fusion and dataflow optimization within compound operations, they do not model the costs associated with collective operations explicitly (Fig. 1), which become critical as the size of compound operations increases and workloads are distributed across spatial clusters or compute cores. Furthermore, the cost models used in these frameworks are derived from single-operator performance models such as Timeloop [20], which do not account for data staging inefficiencies—such as the data transfer time from a parent memory to a child memory when the compute has not started (ramp-up phase). These latencies become particularly important when modeling compound operations due to fine-grained inter-operation dependencies.

To address these challenges, we propose COMET, a framework for modeling the dataflows of compound operations with a novel, explicit representation for collective operations. We further develop a cost model that incorporates the impact of data staging inefficiencies on the performance and scheduling of compound operations.

IV COMET

The overall workflow of COMET is shown in Fig. 3. The user provides three inputs to COMET : the workload description for the compound operation, the description of the underlying hardware architecture, and a mapping for the given compound operation. The mapping, along with the specified constraints (to narrow down the search space) and workload description, is used by the mapping instance generator to select a valid set of tiling factors, loop ordering and spatial unrolling dimensions, thereby generating a concrete mapping instance. Before constructing COMET’s intermediate representation (IR), a validation check is performed to ensure that all tensors fit within the memory hierarchy of the target architecture. Once validated, the mapping instance is converted into an IR, which captures both the data movement and reuse patterns across the memory hierarchy. This representation, including collectives, is then used by the cost model to estimate the latency and energy of the compound operation.

IV-A Explicit Collective Representation

We characterize the dataflow for compound operation as a four-dimensional (4D) design space, as shown in Fig. 1. The first design choice involves selecting the tiling factors, loop ordering and spatial unrolling strategies for all the tensors used in the compound operation. The second design choice is the selection of explicit collective operations required for the correct execution of the compound operation, which depends on the underlying hardware architecture and the dataflow. The third design choice considers the execution order of all elementary and collective operations, as well as the level in the memory hierarchy where each collective operation is performed. Finally, the fourth design choice addresses the scheduling of multiple compute and memory operations across the available hardware resources.

Spatial accelerators, such as the one shown in Fig. 2(b), require modeling two types of communication to accurately capture data movement and synchronization overheads during distributed execution of compound operations:

First, parent-to-child memory communication (e.g. DRAM $\rightarrow$ GB). To represent this, we adopt a loop nest based structure similar to previous works [20, 32]. However, we extend this representation by assigning each tensor a unique loop nest per memory level. This allows for precise modeling of data movement and reuse in compound operations and supports hardware configurations where different tensors are stored in separate memory instances (more results on this in Section V-B). These are represented by tile nodes ($T_{i}^{j}$), where $i$ denotes the memory level and $j$ is the index of the tile node.

Second, peer-to-peer communication between memories at the same hierarchy level (e.g. GBcluster0 $\leftrightarrow$ GBcluster1) and is used to synchronize or redistribute data across spatially partitioned cores or clusters. We explicitly model these interactions using collective operation nodes ($CO_{i}^{j}$), enabling us to capture and analyze communication patterns that were implicit or unsupported in earlier frameworks. Each collective operation node is annotated with the following attributes:

• •

ColOpType: Specifies the type of collective operation to be performed.

• •

Tensor: Indicates the tensor on which the collective operation is applied.

• •

ReduceOp: Defines the arithmetic function (e.g., max, add) used in reduction collectives.

• •

Src and Dest: Lists the source and destination memory levels involved in the collective operation. These lists allow us to generalize collective operations occurring at different memory hierarchy levels. For example, Src = [GB, OB] and Dest = [OB] represents a collective operation occurring between all OBs in the architecture. Src=[OB] and Dest = [OB] represents a collective operation between all OBs within the cluster.

To illustrate the expressiveness of our IR, consider the compound operation shown in Fig. 4(a): a GEMM followed by Softmax, where the Softmax is decomposed into sub-operations ($Op3$ to $Op7$), following a distributed softmax structure similar to [6]. Fig. 4(b) presents a sample mapping where the $N$ dimension is spatially partitioned across clusters. The full mapping is expressed as a hierarchical tree structure (Fig. 4(c)) with different operations fused at different level of the memory hierarchy.

When a tile node has multiple children, it indicates that multiple operations are fused at that memory level. In such cases, COMET supports user-specified scheduling strategies—such as sequential, pipelined or parallel execution—to model the temporal ordering and resource sharing between fused operations.

Execution begins at the root node $T_{0}^{0}$, representing data movement from DRAM to the GBs. While this may internally involve a broadcast, it is treated as implicit for simplicity. The children of $T_{1}^{0}$ correspond to different elementary operations fused at the GB level. Nodes $T_{2}^{0}$ and $T_{2}^{1}$ execute the GEMM and non-GEMM operations respectively on dedicated compute units, and the result is written back through the OB and GB. The next elementary operation, $Op4$ (element-wise subtraction), requires access to the maximum value across the full row of tensor $C$. Since the tensor $C$ is spatially partitioned along the $N$ dimension, an All-Reduce collective operation must be performed to compute the correct maximum value across clusters. This collective operation is explicitly represented using the proposed explicit collective representation ($CO_{1}^{0}$). Next, Op4 and Op5 are fused at OB level. Subsequently, another All-Reduce, represented by $CO_{1}^{1}$, is invoked to compute the softmax denominator. Finally, $Op6$ and $Op7$ are fused at OB level and executed on the SIMD unit to get the final output.

This tree-based explicit collective representation enables exploration of alternative mappings that trade off memory usage and communication overhead. For example, after executing node $T_{1}^{1}$, the intermediate output tensor $C$ is spatially distributed across clusters. However, subsequent operations like Softmax require access to the full row of tensor $C$ (i.e. all columns). Instead of performing an All-Reduce to compute the required reductions across clusters, a gather collective operation can be used to move all columns of a row to a single cluster. This enables the entire Softmax—including element-wise operations like subtraction, division, and exponentiation—to be executed locally, avoiding repeated cross-cluster synchronizations. While this reduces communication overhead, it requires more local memory to store the gathered tensor. Alternatively, collective operations can be moved to lower levels in the memory hierarchy (e.g. OB level) to reduce write-back costs.

Thus, COMET ’s explicit collective representation enables flexible design tradeoffs: it lets users shift when and where collective operations are applied to balance memory pressure, communication cost, and compute locality based on the hardware and workload.

IV-B Compound Operation Performance Analysis

Energy Model: We use an access-count-based energy model similar to [14], where the number of accesses to different hardware components is calculated based on the mapping and data reuse patterns. The total energy is then computed as the sum of the energy consumed by all individual components. The NoC energy is estimated using the NoC energy model from [29], which is based on the Orion power model [13].

Memory Transaction Latency: We assume that all on-chip memories are double buffered and dual port, consistent with prior analytical cost models [20, 32]. This assumption enables overlapping memory transactions with compute operations. A typical timing diagram for a memory transfer from a parent memory (PM) to a child memory (CM) is shown in Fig. 5. The memory transaction latency for single iteration is estimated similar to [20, 32] shown in Eq. 1, where DV denotes the data volume moved per iteration and BW is the memory bandwidth.

[TABLE]

The total latency for moving a tensor volume for $N$ iterations from a parent memory to child memory is given by Eq. 2:

[TABLE]

Here, $MW$ (memory window) is the time required for the parent memory to continuously fill the child memory without any stalls. If child memory is a compute node, $MW$ corresponds to compute time; if child memory is a non-leaf node, it corresponds to the latency of that node. The compulsory stall ($\operatorname{CS}$) accounts for the initial fill or final drain of data, while the optional stall ($\operatorname{OS}$) captures additional latency incurred when the memory transaction time (Eq. 1) exceeds the memory window.

The total latency for a given mapping is computed using this iteration-level model at each node in the mapping tree, starting from the leaf nodes and progressing hierarchically upward to the root node. This analysis also captures operation-level dependencies: for instance, in Fig. 4(c), node $T_{1}^{2}$ must wait for $T_{1}^{1}$ to write data back to the global buffer (GB) before starting execution, if a sequential scheduling strategy exists between them. Such dependencies introduce compulsory stalls that must be accounted for when analyzing the performance of compound operations.

Collective Operation Latency: The memory latency for collective operation is estimated using Eq. 1, where the effective bandwidth is limited by the NoC channel bandwidth. The NoC latency is estimated using the analytical model proposed in [29], as shown in Eq. 3, where $DV$ is the total data volume moved during the collective operation, $W$ is the NoC channel width, $hops$ is the total number of hops for the collective operation, $t_{router}$ is the latency associated with the routing computation, switch allocation and $t_{enq}$ represents the queuing delay. We use the recursive doubling and halving algorithm [30] to compute both the total number of hops and the total data volume moved for different types of collective operations.

[TABLE]

The total latency of a collective operation node is defined in Eq. 4, where $\operatorname{MemLat}$ denotes the memory access latency and $\operatorname{NoCLat}$ denotes the corresponding NoC latency for the memory module involved in the collective operation.

[TABLE]

Latency with Scheduling: If a node $T_{n}^{0}$ has two children $T_{n-1}^{0}$ and $T_{n-1}^{1}$, its latency is computed as shown in Eq. 5. In contrast for pipelied or parallel scheduling, the latency is determined by the slower child, with an additional penalty if both children contend for the same memory resource. This penalty, referred to as conflict stall, is computed using Eq.6. The actual stall time arises only when there is contention and its value is given by Eq.7. Here, $\operatorname{MemLat}(T_{n-1}^{i})$ refers to the latency of the shared memory accessed by both child nodes.

[TABLE]

[TABLE]

[TABLE]

V Evaluation

V-A Methodology

We developed the COMET simulator in C++ to estimate the latency and energy consumption of different compound operations across diverse dataflows. The simulator accepts YAML-formatted specifications of the workload, mapping, architecture description and mapping constraints, similar to previous works [20, 32].

Workloads: To demonstrate the importance of COMET for performance analysis, we study three compound operations: GEMM-Softmax (GEMM-SM) and GEMM-LayerNorm (GEMM-LN) and self-attention (Attn) [27]. We consider different GEMM shapes derived from the Llama-3 and GPT-2 models [26, 22], covering both prefill and decode phases, as shown in Tables I–IV.

Hardware Modeling: We evaluated the performance of these compound operations under two accelerator configurations, using the architecture shown in Fig. 2(b) as a template, targeting the edge and cloud deployment scenarios described in Table V. The DRAM read/write energy is estimated using the DRAMPower tool [4] for DDR4 memory. The energy and latency of the on-chip memories are estimated using CACTI version 7 [18]. We use $8\times 8$ spatial grid of systolic array as the GEMM unit in every core [23], where each array is of size $32\times 32$. The latency of the systolic array is modeled using the analytical equations from SCALE-Sim [23], and the energy is taken from [29]. The energy and latency of the SIMD units for non-GEMM operations are estimated using Synopsys DesignWare IPs synthesized at a 65nm process node and scaled to 32nm using predictive technology models [25].

Map space search: Our evaluation aims to assess the effectiveness of the proposed mapping representation and cost model in capturing the performance trade-offs of compound operations. This is achieved by performing an iterative search over the map space for each workload to identify dataflows that minimize latency. While more advanced search techniques could be used, our goal is not to optimize the search itself, but rather to demonstrate the utility of the representation and cost model in guiding dataflow decisions. The framework allows user to specify constraints to prune the search space, enabling the framework to efficiently identify optimal or near optimal dataflows. In our experiments, we use up to 10,000 search iterations for compound operations. Although we currently employ an iterative search approach, the framework can be easily augmented with alternative search algorithms, which we leave as future work.

V-B Cost Model Comparison

We evaluate our energy and latency cost models by comparing them against Timeloop [20] for single operations and TileFlow [32] for compound operations. For both comparisons, we adopt the same hardware architecture used in TileFlow, comprising DRAM, an on-chip buffer, and a MAC array. In the single operation case, we use a GEMM workload and evaluate 1152 different mappings. For the compound operation case, we consider a GEMM-GEMM sequence, where the intermediate output is forwarded from the first to the second operation.

Timeloop: Fig. 6(a) shows the inference energy comparison for single operations. Our model exhibits a strong correlation with Timeloop, which is expected since both frameworks estimate energy based on component level access counts. Fig. 6(b) presents the latency comparison. While correlation remains high, COMET tends to estimate slightly higher latency than Timeloop. This discrepancy arises because our model accounts for data staging inefficiencies, specifically the ramp-up and ramp-down costs at each iteration. In contrast, Timeloop assumes steady-state operation with perfect pipelining, thus underestimating the latency in some scenarios.

TileFlow: Fig. 6(c) presents the inference energy comparison for compound operations. Our estimates show high correlation with TileFlow, however, COMET tends to report slightly lower energy values. This is primarily because TileFlow does not account for intermediate tensor reuse when operations are fused at certain levels – a limitation acknowledged in Section 7.1 of the TileFlow paper [32]. COMET captures such reuse through its mapping representation (Section IV-A), which assigns a unique loop nest to each tensor at every level of the memory hierarchy. Fig. 6(d) shows the corresponding latency comparison. Again, while the overall correlation is high, COMET produces higher latency estimates. This is due to two factors: i) our model includes the ramp-up and ramp-down effects, and ii) we explicitly model operation-level dependencies. In particular, the second GEMM can only begin execution after the first has completed writing its output from the MAC array to the on-chip buffer, introducing compulsory stalls not modeled by TileFlow.

V-C Case Studies

Next, we present case studies for the GEMM-Softmax and GEMM-LayerNorm compound operations to highlight the importance of our explicit collective operation representation introduced in Section IV-A. For each case, we consider two mapping strategies based on how the non-GEMM operation is scheduled across the hardware hierarchy.

For the GEMM-Softmax case, we refer to the mapping shown in Fig. 4(c) as distributed Softmax (distSM), where both GEMM and Softmax operations are spatially distributed across multiple clusters and cores. In contrast, the standard Softmax mapping (SM) distributes only the GEMM across clusters and cores, while the Softmax is executed entirely within a single cluster and core. This avoids the need for a costly All-Reduce operation; instead, a simpler gather operation is performed after the $T_{1}^{1}$ node in Fig.4(c).

A similar mapping strategy for the GEMM-LayerNorm compound operation can be constructed using the representation shown in Fig. 4(c), although different non-GEMM operations are involved. The GEMM-LayerNorm operation also requires two All-Reduce collectives, but across different tensor shapes. We again define two mapping types: distributed LayerNorm (distLN) and standard LayerNorm (LN). Due to limited space, we avoid explicitly showing the full mapping tree for GEMM-LayerNorm.

V-C1 Latency and Energy Variation

Fig. 7 shows the latency and energy results for the different dataflow mappings of GEMM-Softmax and GEMM-LayerNorm operations. We observe that the choice between distributed (distSM, distLN) and non-distributed (SM, LN) mappings significantly impacts both latency and energy consumption. In particular, we find that non-distributed mappings sometimes encounter out-of-memory (OOM) scenarios. Next, we analyze these trends by examining the detailed latency and energy breakdown for different workloads listed in TableI and Table II.

V-C2 Latency Breakdown Analysis

For this analysis, we follow the mapping representation from Fig. 4(c), where dimension $N$ is spatially mapped across clusters and cores, while dimension $M$ is temporally mapped. Essentially, this corresponds to FLAT’s row-granularity dataflow [14].

We observe that for larger GEMM operations (GEMM9, GEMM11, GEMM12), the total latency in Fig. 8(a) is dominated by the SIMD unit for the SM mapping. This is because the Softmax operation is executed entirely on a single core. In contrast, for the distSM mapping, the latency is dominated by the collective operation overhead. This occurs because a large $M$ dimension (in GEMM9, GEMM11, GEMM12) results in frequent collective operations being triggered. For workloads with smaller $M$ values (e.g., GEMM1, GEMM2, GEMM4, GEMM7, GEMM8), latency is instead dominated by compulsory stalls (CS), due to lower data reuse opportunities in these cases.

For the GEMM-LayerNorm compound operation (Fig. 8(b)), we observe that latency in the distLN mapping is dominated by compulsory stalls across all workloads. This is because, in distLN, the collective operations are performed on smaller tensors (of size $M\times 1$) compared to distSM, where the collectives operate on larger $M\times N$ tensors. Similarly, for GEMMs with larger $M$ values under the LN mapping, the latency is dominated by the SIMD unit execution.

V-C3 Energy Breakdown Analysis

When moving from distributed to standard Softmax/LayerNorm mappings, the number of accesses to hardware architecture components remains largely unchanged; only the type of collective operation changes. However, the overall energy consumption is still dominated by off-chip memory accesses, due to the high read and write energy of DRAM. For larger GEMM operations, we observe that collective operations begin to contribute noticeably to the total energy, as shown in Fig. 9(a,b).

V-D Fusion Mapping Comparison

V-D1 GEMM-Softmax and GEMM-LayerNorm

We compare different fusion mappings for GEMM-Softmax and GEMM-LayerNorm compound operations, as shown in Fig.10. The Unfused mapping refers to executing all elementary operations sequentially, with data read from and written back to DRAM between each operation. In the Fused-distSM mapping, the elementary operations of Softmax are fused together, but not with the preceding GEMM operation. In the Fused-GEMM-SM mapping, the GEMM and Softmax are fused, but the Softmax computation is restricted to a single cluster and core (standard mapping). Finally, Fused-GEMM-distSM represents the mapping where all elementary operations in the compound operation GEMM-Softmax are fused (as shown in Fig.4(c)). Similar terminology is used for the GEMM-LayerNorm compound operation.

We observe that different compound operations benefit from different fusion strategies:

• •

All fused mappings exhibit lower latency compared to the unfused baseline, primarily due to reduced off-chip memory accesses for intermediate tensors. We observe a geometric mean latency reduction of 1.42$\times$ for GEMM-Softmax and 3.46$\times$ for GEMM-LayerNorm.

• •

Comparing Fused-GEMM-distSM with Fused-distSM in Fig. 10(a), we observe that Fused-GEMM-distSM consistently achieves lower latency. This is because fusing all elementary operations in the compound operation eliminates additional memory transfers between operations.

• •

For the edge platform (GEMM1–GEMM6), Fused-GEMM-SM exhibits higher latency than other fused mappings. This is because, as shown in Fig. 8(a), the latency in SM mappings is dominated by the SIMD unit, leading to overall higher execution time. However, for the cloud platform (GEMM7–GEMM12), collective operations start dominating the latency in distSM mappings (Fig. 8(a)). Thus, Fused-GEMM-SM can reduce collective operation latency by executing Softmax on a single cluster and core.

• •

For the GEMM-LayerNorm compound operation (Fig. 10(b)), Fused-GEMM-distLN consistently achieves the lowest latency. The underlying reason is similar to the GEMM-Softmax case; however, the improvement is even greater because LayerNorm involves more elementary operations being fused than Softmax.

• •

In contrast, Fused-GEMM-LN consistently exhibits higher latency than other fused mappings. This is because the distLN mapping is not bottlenecked by collective operation latency, and the latency associated with the SIMD unit in the Fused-GEMM-LN mapping is significantly higher, as shown in Fig. 8(b).

Fig. 11 shows the energy results for different fusion mappings. All fusion mappings exhibit lower energy consumption compared to the unfused baseline, primarily due to reduced DRAM accesses in the fused configurations. Moreover, Fused-GEMM-distSM and Fused-GEMM-SM mappings show only a small difference in total energy, since the number of memory accesses remains largely unchanged when moving from distributed to non-distributed mappings.

V-D2 Self-Attention

We compare three different variants of the self-attention operation. Unfused Attention (UA) executes the score ($Q\cdot K^{\top}$), softmax and context ($\text{softmax}(Q\cdot K^{\top})\cdot V$) operations independently. Partially Fused Attention (PFA) fuses the score and softmax operations while keeping the context operation separate. Flash Attention (FA) [6] fuses all three operations and employs a distributed softmax to optimize memory access and dataflow. These variants exhibit different latency and energy characteristics depending on the attention operation and platform.

• •

As shown in Fig. 12(a,b) fused attention variants (PFA and FA) consistently achieve lower latency and energy compared to the unfused baseline. This improvement is primarily due to reduced data movement of the intermediate tensors. We observe a geometric mean latency reduction of 1.82$\times$ and a geometric mean energy reduction of 1.54$\times$ using FA.

• •

On the edge platform, for attention operations with limited reuse (e.g. Attn2, Attn3, Attn4), latency improvements from fusion are less pronounced. As seen in Fig. 13, latency is dominated by DRAM access, and the small size of intermediate tensors limit fusion benefits.

• •

On the cloud platform, for similar low-reuse workloads (e.g. Attn8, Attn9, Attn10), the latency benefits from PFA and FA are more significant. This is because the larger intermediate tensor sizes on the cloud platform provide greater opportunity for fusion to reduce data movement overhead.

• •

For other operations on both the edge and cloud platforms, we observe substantial latency improvements with fused operations. However, an interesting observation is that transitioning from UA to FA increases the latency of SIMD units. This is due to additional non-GEMM computations (Fig. 2(a)) introduced by the Flash Attention algorithm to enable the fusion of score, softmax and context operations. On the other hand, collective operation latency decreases, primarily because fusion reduces the number of implicit collectives by minimizing off-chip memory accesses.

• •

Both PFA and FA leads to reduced energy consumption compared to UA. As shown in Fig. 14, FA has lower off-chip memory access energy compared to UA and PFA, resulting in the energy improvements observed in Fig. 12(b).

These results highlight the importance of selecting appropriate fusion and collective strategies based on the operation structure and hardware configuration, as they significantly impact data reuse, memory footprint and overall performance. Additionally, by accurately modeling the latency of non-GEMM operations in conjunction with their dependencies on GEMM computations, COMET provides insights that can guide future optimizations of non-GEMM compute units, complementing recent efforts such as [7].

VI Conclusion

In this paper, we proposed COMET, a framework for modeling and optimizing dataflows for compound operations deployed on machine learning accelerators. Modern DNNs frequently rely on compound operations, which introduce new challenges in memory access patterns, communication overhead, and dataflow optimization. To address these challenges, COMET introduces an explicit collective representation to map the compound operation on ML accelerators. It also introduces a cost model that captures both compute and communication latencies, including compulsory and optional memory stalls. We demonstrated that different mapping strategies enabled by our explicit collective representation significantly impact the latency and energy efficiency of compound operations such as GEMM-Softmax, GEMM-LayerNorm and self-attention. Through detailed case studies on both the edge and cloud accelerator platforms, we demonstrated that fusion strategies and the choice of collective operations play a critical role in balancing compute and communication costs. Our optimized dataflows achieve a geometric mean speedup of 1.42$\times$ for GEMM-Softmax, 3.46$\times$ for GEMM-LayerNorm and 1.82$\times$ for self-attention compared to unfused baselines.

Acknowledgment

This work was supported in part by the Center for the Co-Design of Cognitive Systems (COCOSYS), a DARPA-sponsored JUMP center of Semiconductor Research Corporation (SRC), in part by the National Science Foundation, in part by Intel Corporation and in part by the Department of Energy. The authors would also like to thank Deepika Sharma for her feedback on the manuscript and Zhenyu Wang for his assistance in understanding the NoC model from HISIM.