Efficient Unified Caching for Accelerating Heterogeneous AI Workloads

TL;DR

This paper introduces IGTCache, a unified caching framework for heterogeneous AI workloads that detects access patterns to optimize cache management, significantly improving cache hit ratio and reducing job completion time.

Contribution

It presents a novel hierarchical access abstraction and pattern detection method to enable a unified cache management strategy for diverse AI workloads.

Findings

Increases cache hit ratio by 55.6%

Reduces job completion time by 52.2%

Effectively handles heterogeneous access patterns

Abstract

Modern AI clusters, which host diverse workloads like data pre-processing, training and inference, often store the large-volume data in cloud storage and employ caching frameworks to facilitate remote data access. To avoid code-intrusion complexity and minimize cache space wastage, it is desirable to maintain a unified cache shared by all the workloads. However, existing cache management strategies, designed for specific workloads, struggle to handle the heterogeneous AI workloads in a cluster -- which usually exhibit heterogeneous access patterns and item storage granularities. In this paper, we propose IGTCache, a unified, high-efficacy cache for modern AI clusters. IGTCache leverages a hierarchical access abstraction, AccessStreamTree, to organize the recent data accesses in a tree structure, facilitating access pattern detection at various granularities. Using this abstraction, IGTCache applies hypothesis testing to categorize data access patterns as sequential, random, or skewed. Based on these detected access patterns and granularities, IGTCache tailors optimal cache management strategies including prefetching, eviction, and space allocation accordingly. Experimental results show that IGTCache increases the cache hit ratio by 55.6% over state-of-the-art caching frameworks, reducing the overall job completion time by 52.2%.