Objcache: An Elastic Filesystem over External Persistent Storage for Container Clusters

TL;DR

Objcache is an elastic filesystem designed for container clusters that efficiently manages caching over external storage, significantly reducing startup times and supporting scalable, fault-tolerant AI workloads.

Contribution

It introduces a novel transaction protocol over Raft for consistent distributed cache management and supports elastic scaling down to zero with automatic eviction.

Findings

Speeded up model serving startup by 98.9%

Managed scaling with up to 1024 dirty files in seconds

Enabled elastic scaling down to zero with automatic eviction

Abstract

Container virtualization enables emerging AI workloads such as model serving, highly parallelized training, machine learning pipelines, and so on, to be easily scaled on demand on the elastic cloud infrastructure. Particularly, AI workloads require persistent storage to store data such as training inputs, models, and checkpoints. An external storage system like cloud object storage is a common choice because of its elasticity and scalability. To mitigate access latency to external storage, caching at a local filesystem is an essential technique. However, building local caches on scaling clusters must cope with explosive disk usage, redundant networking, and unexpected failures. We propose objcache, an elastic filesystem over external storage. Objcache introduces an internal transaction protocol over Raft logging to enable atomic updates of distributed persistent states with consistent hashing. The proposed transaction protocol can also manage inode dirtiness by maintaining the consistency between the local cache and external storage. Objcache supports scaling down to zero by automatically evicting dirty files to external storage. Our evaluation reports that objcache speeded up model serving startup by 98.9% compared to direct copies via S3 interfaces. Scaling up with dirty files completed from 2 to 14 seconds with 1024 dirty files.