Rearchitecting Kubernetes for the Edge

TL;DR

This paper proposes an eventually consistent approach to Kubernetes to improve performance, availability, and scalability at the edge by addressing the latency and throughput issues caused by etcd's strong consistency.

Contribution

It introduces an alternative consistency model for Kubernetes, enhancing its suitability for edge environments with performance-critical and scalable needs.

Findings

Higher etcd cluster sizes increase write latency and decrease throughput.

Approximately 30% of Kubernetes requests are writes, impacting latency and availability.

Eventual consistency improves performance and scalability for edge deployments.

Abstract

Recent years have seen Kubernetes emerge as a primary choice for container orchestration. Kubernetes largely targets the cloud environment but new use cases require performant, available and scalable orchestration at the edge. Kubernetes stores all cluster state in etcd, a strongly consistent key-value store. We find that at larger etcd cluster sizes, offering higher availability, write request latency significantly increases and throughput decreases similarly. Coupled with approximately 30% of Kubernetes requests being writes, this directly impacts the request latency and availability of Kubernetes, reducing its suitability for the edge. We revisit the requirement of strong consistency and propose an eventually consistent approach instead. This enables higher performance, availability and scalability whilst still supporting the broad needs of Kubernetes. This aims to make Kubernetes much more suitable for performance-critical, dynamically-scaled edge solutions.