Data Dwarfs: A Lens Towards Fully Understanding Big Data and AI Workloads

TL;DR

This paper introduces eight fundamental data dwarfs that capture the core computational patterns of big data and AI workloads, enabling better understanding and benchmarking of these complex systems.

Contribution

It identifies eight key data dwarfs representing major workload components and implements them as micro benchmarks for comprehensive analysis.

Findings

Eight data dwarfs account for most runtime in big data and AI workloads.

Implementation of data dwarfs as micro benchmarks facilitates workload characterization.

Provides insights into data sizes, types, sources, and patterns in workloads.

Abstract

The complexity and diversity of big data and AI workloads make understanding them difficult and challenging. This paper proposes a new approach to characterizing big data and AI workloads. We consider each big data and AI workload as a pipeline of one or more classes of unit of computations performed on different initial or intermediate data inputs. Each class of unit of computation captures the common requirements while being reasonably divorced from individual implementations, and hence we call it a data dwarf. For the first time, among a wide variety of big data and AI workloads, we identify eight data dwarfs that takes up most of run time, including Matrix, Sampling, Logic, Transform, Set, Graph, Sort and Statistic. We implement the eight data dwarfs on different software stacks as the micro benchmarks of an open-source big data and AI benchmark suite, and perform comprehensive characterization of those data dwarfs from perspective of data sizes, types, sources, and patterns as a lens towards fully understanding big data and AI workloads.