HPTMT Parallel Operators for High Performance Data Science & Data Engineering

TL;DR

This paper introduces the HPTMT architecture, a unified framework of data structures and operators designed to enhance the efficiency and interoperability of data science and engineering applications.

Contribution

It proposes a comprehensive architecture that unifies data structures and operators across data engineering and data science, enabling more efficient and integrated applications.

Findings

Demonstrates the architecture with an end-to-end application

Shows improved efficiency in data processing and integration

Provides a clear framework linking data engineering and science

Abstract

Data-intensive applications are becoming commonplace in all science disciplines. They are comprised of a rich set of sub-domains such as data engineering, deep learning, and machine learning. These applications are built around efficient data abstractions and operators that suit the applications of different domains. Often lack of a clear definition of data structures and operators in the field has led to other implementations that do not work well together. The HPTMT architecture that we proposed recently, identifies a set of data structures, operators, and an execution model for creating rich data applications that links all aspects of data engineering and data science together efficiently. This paper elaborates and illustrates this architecture using an end-to-end application with deep learning and data engineering parts working together.