Exploiting Kubernetes' Image Pull Implementation to Deny Node Availability

TL;DR

This paper reveals a security vulnerability in Kubernetes' image pull process that can be exploited to cause denial of service, and proposes an eBPF-based mitigation to detect and stop such attacks.

Contribution

The paper identifies a novel security flaw in Kubernetes' CRI-API handling of image downloads and introduces MAGI, an eBPF-based tool to mitigate this vulnerability.

Findings

Attack can cause up to 95% CPU usage on nodes

Attack prevents new container image downloads

MAGI effectively detects and terminates attack processes

Abstract

Kubernetes (K8s) has grown in popularity over the past few years to become the de-facto standard for container orchestration in cloud-native environments. While research is not new to topics such as containerization and access control security, the Application Programming Interface (API) interactions between K8s and its runtime interfaces have not been studied thoroughly. In particular, the CRI-API is responsible for abstracting the container runtime, managing the creation and lifecycle of containers along with the downloads of the respective images. However, this decoupling of concerns and the abstraction of the container runtime renders K8s unaware of the status of the downloading process of the container images, obstructing the monitoring of the resources allocated to such process. In this paper, we discuss how this lack of status information can be exploited as a Denial of Service attack in a K8s cluster. We show how such attacks can impact worker nodes, generating up to 95% average CPU usage, prevent downloads of new container images, and increase I/O and network usage for a potentially unlimited amount of time. We argue that solving this problem would require a radical architectural change in the relationship between K8s and the CRI-API, which would be unfeasible in the short term. Thus, as a stopgap solution, we propose MAGI: an eBPF-based, proof-of-concept mitigation that detects and terminates potential attacks.