Call Scheduling to Reduce Response Time of a FaaS System

TL;DR

This paper introduces a worker-level scheduling method for FaaS systems that significantly reduces response times under heavy loads without adding more nodes, by queuing requests based on history and limiting CPU usage per request.

Contribution

The paper proposes a novel scheduling approach inspired by HPC techniques, replacing OS preemption with request queuing and CPU limiting, improving FaaS response times under load.

Findings

Average response time decreased by a factor of 4.

Average request stretch decreased by a factor of 18.

Fewer machines needed for better response-time statistics.

Abstract

In an overloaded FaaS cluster, individual worker nodes strain under lengthening queues of requests. Although the cluster might be eventually horizontally-scaled, adding a new node takes dozens of seconds. As serving applications are tuned for tail serving latencies, and these greatly increase under heavier loads, the current workaround is resource over-provisioning. In fact, even though a service can withstand a steady load of, e.g., 70% CPU utilization, the autoscaler is triggered at, e.g., 30-40% (thus the service uses twice as many nodes as it would be needed). We propose an alternative: a worker-level method handling heavy load without increasing the number of nodes. FaaS executions are not interactive, compared to, e.g., text editors: end-users do not benefit from the CPU allocated to processes often, yet for short periods. Inspired by scheduling methods for High Performance Computing, we take a radical step of replacing the classic OS preemption by (1) queuing requests based on their historical characteristics; (2) once a request is being processed, setting its CPU limit to exactly one core (with no CPU oversubscription). We extend OpenWhisk and measure the efficiency of the proposed solutions using the SeBS benchmark. In a loaded system, our method decreases the average response time by a factor of 4. The improvement is even higher for shorter requests, as the average stretch is decreased by a factor of 18. This leads us to show that we can provide better response-time statistics with 3 machines compared to a 4-machine baseline.