Increasing Efficiency and Result Reliability of Continuous Benchmarking for FaaS Applications

TL;DR

DuetFaaS enhances continuous benchmarking for FaaS by deploying two function versions simultaneously, reducing invocation needs and improving detection accuracy of performance regressions amidst platform variability.

Contribution

The paper introduces DuetFaaS, a novel duet benchmarking method for FaaS that significantly reduces the number of invocations needed for reliable performance regression detection.

Findings

Requires fewer invocations than existing methods

Achieves smaller confidence intervals in most cases

Reduces interval size in over half of the evaluations

Abstract

In a continuous deployment setting, Function-as-a-Service (FaaS) applications frequently receive updated releases, each of which can cause a performance regression. While continuous benchmarking, i.e., comparing benchmark results of the updated and the previous version, can detect such regressions, performance variability of FaaS platforms necessitates thousands of function calls, thus, making continuous benchmarking time-intensive and expensive. In this paper, we propose DuetFaaS, an approach which adapts duet benchmarking to FaaS applications. With DuetFaaS, we deploy two versions of FaaS function in a single cloud function instance and execute them in parallel to reduce the impact of platform variability. We evaluate our approach against state-of-the-art approaches, running on AWS Lambda. Overall, DuetFaaS requires fewer invocations to accurately detect performance regressions than other state-of-the-art approaches. In 98.41% of evaluated cases, our approach provides equal or smaller confidence interval size. DuetFaaS achieves an interval size reduction in 59.06% of all evaluated sample sizes when compared to the competitive approaches.