OpenIVM: a SQL-to-SQL Compiler for Incremental Computations

TL;DR

OpenIVM introduces a SQL-based incremental view maintenance system that leverages existing SQL engines like DuckDB, enabling seamless integration and cross-system HTAP workflows without additional code complexity.

Contribution

It presents a novel SQL-to-SQL compiler for IVM that integrates with existing engines, simplifying incremental view maintenance and cross-system data propagation.

Findings

OpenIVM effectively integrates with DuckDB and PostgreSQL.

It enables cross-system IVM in HTAP architectures.

Demonstrated successful implementation as a DuckDB extension.

Abstract

This demonstration presents a new Open Source SQL-to-SQL compiler for Incremental View Maintenance (IVM). While previous systems, such as DBToaster, implemented computational functionality for IVM in a separate system, the core principle of OpenIVM is to make use of existing SQL query processing engines and perform all IVM computations via SQL. This approach enables the integration of IVM in these systems without code duplication. Also, it eases its use in cross-system IVM, i.e. to orchestrate an HTAP system in which one (OLTP) DBMS provides insertions/updates/deletes (deltas), which are propagated using SQL into another (OLAP) DBMS, hosting materialized views. Our system compiles view definitions into SQL to eventually propagate deltas into the table that materializes the view, following the principles of DBSP. Under the hood, OpenIVM uses the DuckDB library to compile (parse, transform, optimize) the materialized view maintenance logic. We demonstrate OpenIVM in action (i) as the core of a DuckDB extension module that adds IVM functionality to it and (ii) powering cross-system IVM for HTAP, with PostgreSQL handling updates on base tables and DuckDB hosting materialized views on these.