Reducing Load Latency with Cache Level Prediction

TL;DR

This paper introduces a cache-level prediction method that accurately forecasts memory hierarchy levels accessed by loads, enabling earlier data fetching and reducing load latency, resulting in notable performance improvements.

Contribution

It presents a novel cache-level predictor that complements prefetchers by accurately predicting memory hierarchy levels accessed, improving load latency and overall performance.

Findings

Achieves 7.8% speedup on various applications.

Provides high prediction accuracy with minimal added latency.

Effectively reduces load latency in deep cache hierarchies.

Abstract

High load latency that results from deep cache hierarchies and relatively slow main memory is an important limiter of single-thread performance. Data prefetch helps reduce this latency by fetching data up the hierarchy before it is requested by load instructions. However, data prefetching has shown to be imperfect in many situations. We propose cache-level prediction to complement prefetchers. Our method predicts which memory hierarchy level a load will access allowing the memory loads to start earlier, and thereby saves many cycles. The predictor provides high prediction accuracy at the cost of just one cycle added latency to L1 misses. Experimental results show speedup of 7.8\% on generic, graph, and HPC applications over a baseline with aggressive prefetchers.