Functional Collection Programming with Semi-Ring Dictionaries

TL;DR

This paper presents semi-ring dictionaries as a versatile, purely functional collection framework, and introduces SDQL, a language that unifies database and linear algebra optimizations for efficient hybrid data processing.

Contribution

The paper's main contribution is the development of semi-ring dictionaries and SDQL, enabling unified, optimized processing of database and linear algebra workloads in a purely functional setting.

Findings

SDQL outperforms state-of-the-art systems in hybrid workloads

Efficient processing of relational and linear algebra data using semi-ring dictionaries

Unified optimizations improve performance across diverse data processing tasks

Abstract

This paper introduces semi-ring dictionaries, a powerful class of compositional and purely functional collections that subsume other collection types such as sets, multisets, arrays, vectors, and matrices. We developed SDQL, a statically typed language that can express relational algebra with aggregations, linear algebra, and functional collections over data such as relations and matrices using semi-ring dictionaries. Furthermore, thanks to the algebraic structure behind these dictionaries, SDQL unifies a wide range of optimizations commonly used in databases (DB) and linear algebra (LA). As a result, SDQL enables efficient processing of hybrid DB and LA workloads, by putting together optimizations that are otherwise confined to either DB systems or LA frameworks. We show experimentally that a handful of DB and LA workloads can take advantage of the SDQL language and optimizations. SDQL can be competitive with or outperforms a host of systems that are state of the art in their own domain: in-memory DB systems Typer and Tectorwise for (flat, not nested) relational data; SciPy for LA workloads; sparse tensor compiler taco; the Trance nested relational engine; and the in-database machine learning engines LMFAO and Morpheus for hybrid DB/LA workloads over relational data.