Helper Without Threads: Customized Prefetching for Delinquent Irregular Loads

TL;DR

This paper introduces an inline software prefetching method that enhances performance for irregular memory loads by inserting customized prefetchers directly into the main thread, avoiding hardware or thread context requirements.

Contribution

It presents a novel inline prefetching technique that mimics helper thread prefetching without needing extra threads or hardware support, improving irregular load performance.

Findings

Up to 2X performance improvement on high-end hardware.

83% speedup over helper thread implementation.

Effective for memory-bound irregular workloads.

Abstract

The growing memory footprints of cloud and big data applications mean that data center CPUs can spend significant time waiting for memory. An attractive approach to improving performance in such centralized compute settings is to employ prefetchers that are customized per application, where gains can be easily scaled across thousands of machines. Helper thread prefetching is such a technique but has yet to achieve wide adoption since it requires spare thread contexts or special hardware/firmware support. In this paper, we propose an inline software prefetching technique that overcomes these restrictions by inserting the helper code into the main thread itself. Our approach is complementary to and does not interfere with existing hardware prefetchers since we target only delinquent irregular load instructions (those with no constant or striding address patterns). For each chosen load instruction, we generate and insert a customized software prefetcher extracted from and mimicking the application's dataflow, all without access to the application source code. For a set of irregular workloads that are memory-bound, we demonstrate up to 2X single-thread performance improvement on recent high-end hardware (Intel Skylake) and up to 83% speedup over a helper thread implementation on the same hardware, due to the absence of thread spawning overhead.