HADAD: A Lightweight Approach for Optimizing Hybrid Complex Analytics Queries (Extended Version)

TL;DR

HADAD is a lightweight, extensible approach that optimizes hybrid complex analytics queries by unifying data management and analytics tasks under a relational model with integrity constraints, improving performance across diverse workloads.

Contribution

It introduces HADAD, a novel unified optimization framework for hybrid RA-LA queries that exploits semantic information and precomputed results without modifying underlying systems.

Findings

Significant performance improvements on diverse workloads.

Effective optimization of hybrid RA-LA queries using semantic reasoning.

Applicable to various platforms without internal modifications.

Abstract

Hybrid complex analytics workloads typically include (i) data management tasks (joins, selections, etc. ), easily expressed using relational algebra (RA)-based languages, and (ii) complex analytics tasks (regressions, matrix decompositions, etc.), mostly expressed in linear algebra (LA) expressions. Such workloads are common in many application areas, including scientific computing, web analytics, and business recommendation. Existing solutions for evaluating hybrid analytical tasks - ranging from LA-oriented systems, to relational systems (extended to handle LA operations), to hybrid systems - either optimize data management and complex tasks separately, exploit RA properties only while leaving LA-specific optimization opportunities unexploited, or focus heavily on physical optimization, leaving semantic query optimization opportunities unexplored. Additionally, they are not able to exploit precomputed (materialized) results to avoid recomputing (part of) a given mixed (RA and/or LA) computation. In this paper, we take a major step towards filling this gap by proposing HADAD, an extensible lightweight approach for optimizing hybrid complex analytics queries, based on a common abstraction that facilitates unified reasoning: a relational model endowed with integrity constraints. Our solution can be naturally and portably applied on top of pure LA and hybrid RA-LA platforms without modifying their internals. An extensive empirical evaluation shows that HADAD yields significant performance gains on diverse workloads, ranging from LA-centered to hybrid.