DFModel: Design Space Optimization of Large-Scale Systems Exploiting Dataflow Mappings

TL;DR

DFModel is a comprehensive framework for optimizing dataflow mappings across multiple system hierarchy levels, significantly improving performance, cost, and power efficiency for large-scale workloads including machine learning and HPC applications.

Contribution

It introduces the first multi-level optimization framework for dataflow mappings, covering memory hierarchy and interconnection networks, validated on diverse workloads and systems.

Findings

DFModel predicts 1.25X better performance on average.

Dataflow architectures improve large language model training performance by 1.52X.

Optimized dataflow mappings achieve 6.13X speedup over previous models.

Abstract

We propose DFModel, a modeling framework for mapping dataflow computation graphs onto large-scale systems. Mapping a workload to a system requires optimizing dataflow mappings at various levels, including the inter-chip (between chips) level and the intra-chip (within a chip) level. DFModel is, to the best of our knowledge, the first framework to perform the optimization at multiple levels of the memory hierarchy and the interconnection network hierarchy. We use DFModel to explore a wide range of workloads on a variety of systems. Evaluated workloads include two state-of-the-art machine learning applications (Large Language Models and Deep Learning Recommendation Models) and two high-performance computing applications (High Performance LINPACK and Fast Fourier Transform). System parameters investigated span the combination of dataflow and traditional accelerator architectures, memory technologies (DDR, HBM), interconnect technologies (PCIe, NVLink), and interconnection network topologies (torus, DGX, dragonfly). For a variety of workloads on a wide range of systems, the DFModel provided a mapping that predicts an average of 1.25X better performance compared to the ones measured on real systems. DFModel shows that for large language model training, dataflow architectures achieve 1.52X higher performance, 1.59X better cost efficiency, and 1.6X better power efficiency compared to non-dataflow architectures. On an industrial system with dataflow architectures, the DFModel-optimized dataflow mapping achieves a speedup of 6.13X compared to non-dataflow mappings from previous performance models such as Calculon, and 1.52X compared to a vendor provided dataflow mapping.