Reify Your Collection Queries for Modularity and Speed!

TL;DR

SQuOpt is a Scala-based deep embedding of collection APIs that enables automatic optimizations like fusion and reordering, significantly improving performance while maintaining code modularity.

Contribution

It introduces SQuOpt, a novel Scala query optimizer that reifies collection operations for analysis and optimization without external tools.

Findings

Achieved up to 12800x speedup in code analysis tasks

Re-implemented Findbugs analyses using SQuOpt

Demonstrated that modular code can be highly optimized for performance

Abstract

Modularity and efficiency are often contradicting requirements, such that programers have to trade one for the other. We analyze this dilemma in the context of programs operating on collections. Performance-critical code using collections need often to be hand-optimized, leading to non-modular, brittle, and redundant code. In principle, this dilemma could be avoided by automatic collection-specific optimizations, such as fusion of collection traversals, usage of indexing, or reordering of filters. Unfortunately, it is not obvious how to encode such optimizations in terms of ordinary collection APIs, because the program operating on the collections is not reified and hence cannot be analyzed. We propose SQuOpt, the Scala Query Optimizer--a deep embedding of the Scala collections API that allows such analyses and optimizations to be defined and executed within Scala, without relying on external tools or compiler extensions. SQuOpt provides the same "look and feel" (syntax and static typing guarantees) as the standard collections API. We evaluate SQuOpt by re-implementing several code analyses of the Findbugs tool using SQuOpt, show average speedups of 12x with a maximum of 12800x and hence demonstrate that SQuOpt can reconcile modularity and efficiency in real-world applications.