Harvesting L2 Caches in Server Processors

TL;DR

This paper proposes a novel cache management scheme that leverages idle L2 caches in modern server processors to improve performance and latency, achieving up to 2X speedup and significant latency reductions.

Contribution

It introduces a logical path for LLC evictions to unused private caches, enhancing resource utilization and system performance in multi-core server processors.

Findings

Up to 2X system performance improvement for single-application runs.

Up to 32% P99 latency reduction for user-facing tasks.

Up to 50% IPC improvement for background jobs.

Abstract

We make three observations in modern processors: (1) LLC capacity is getting larger (up to 1GB); (2) core counts are increasing (up to 128 cores), accumulating a more significant amount of private L2 cache capacity on the chip; and (3) overall processor utilization in the cloud remains very low despite many efforts, leaving many large private caches unused. To enable better use of these beefy processors, we propose to open up a logical path for LLC evictions to unused private caches. In other words, instead of writing LLC evictions back to slow and busy main memory, we send some of them that are still alive up to idle L2 caches to avoid unnecessary long and costly main memory. Our scheme takes the importance of applications (user-facing vs. background), and system load into account to provide each application with a fair share of idle resources. Our results show that we can improve system performance by up to 2X (geomean of 10%) for single-application runs. Also, for mixes of user-facing and background jobs, our scheme improves the P99 latency of user-facing tasks by up to 32% (geomean of 15%), and the IPC of background jobs by up to 50% (geomean of 10%).