Reducing Tail Latencies Through Environment- and Neighbour-aware Thread Management

TL;DR

This paper investigates how CPU overcommitment affects tail latency in microservices and introduces a neighbour-aware threadpool, friendlypool, that dynamically reduces overcommitment to lower maximum latency.

Contribution

It identifies the causes of CPU overcommitment and proposes a new neighbour-aware threadpool to mitigate tail latencies in microservice architectures.

Findings

Friendlypool reduces maximum worker latency by up to 6.7 times.

Using friendlypool decreases throughput by up to 1.4 times.

CPU overcommitment significantly impacts tail latency in microservices.

Abstract

Application tail latency is a key metric for many services, with high latencies being linked directly to loss of revenue. Modern deeply-nested micro-service architectures exacerbate tail latencies, increasing the likelihood of users experiencing them. In this work, we show how CPU overcommitment by OS threads leads to high tail latencies when applications are under heavy load. CPU overcommitment can arise from two operational factors: incorrectly determining the number of CPUs available when under a CPU quota, and the ignorance of neighbour applications and their CPU usage. We discuss different languages' solutions to obtaining the CPUs available, evaluating the impact, and discuss opportunities for a more unified language-independent interface to obtain the number of CPUs available. We then evaluate the impact of neighbour usage on tail latency and introduce a new neighbour-aware threadpool, the friendlypool, that dynamically avoids overcommitment. In our evaluation, the friendlypool reduces maximum worker latency by up to $6.7\times$ at the cost of decreasing throughput by up to $1.4\times$.