Unicorn: Reasoning about Configurable System Performance through the lens of Causality

TL;DR

Unicorn is a novel causal inference-based method for understanding and predicting the performance of highly configurable systems across diverse environments, outperforming existing machine learning approaches in reliability and effectiveness.

Contribution

Unicorn introduces a causal inference framework that captures complex interactions in configurable systems, enabling reliable performance predictions and explanations across different hardware and workloads.

Findings

Unicorn outperforms state-of-the-art methods in performance debugging.

It reliably predicts performance in unseen environments.

It effectively finds near-optimal configurations for diverse systems.

Abstract

Modern computer systems are highly configurable, with the total variability space sometimes larger than the number of atoms in the universe. Understanding and reasoning about the performance behavior of highly configurable systems, over a vast and variable space, is challenging. State-of-the-art methods for performance modeling and analyses rely on predictive machine learning models, therefore, they become (i) unreliable in unseen environments (e.g., different hardware, workloads), and (ii) may produce incorrect explanations. To tackle this, we propose a new method, called Unicorn, which (i) captures intricate interactions between configuration options across the software-hardware stack and (ii) describes how such interactions can impact performance variations via causal inference. We evaluated Unicorn on six highly configurable systems, including three on-device machine learning systems, a video encoder, a database management system, and a data analytics pipeline. The experimental results indicate that Unicorn outperforms state-of-the-art performance debugging and optimization methods in finding effective repairs for performance faults and finding configurations with near-optimal performance. Further, unlike the existing methods, the learned causal performance models reliably predict performance for new environments.