Taking the Leap: Efficient and Reliable Fine-Grained NUMA Migration in User-space

TL;DR

This paper introduces a novel user-space method called page_leap() for efficient, reliable, and high-performance NUMA page migration, addressing limitations of existing Linux-based solutions.

Contribution

The paper presents page_leap(), a new user-space migration technique that asynchronously migrates pages with high performance and correctness, improving upon existing NUMA balancing methods.

Findings

Page_leap() achieves higher migration performance than traditional methods.

It ensures correctness during concurrent writes and handles pooled memory.

Supports both small and huge pages for flexible migration.

Abstract

Modern multi-socket architectures offer a single virtual address space, but physically divide main-memory across multiple regions, where each region is attached to a CPU and its cores. While this simplifies the usage, developers must be aware of non-uniform memory access (NUMA), where an access by a thread running on a core-local NUMA region is significantly cheaper than an access from a core-remote region. Obviously, if query answering is parallelized across the cores of multiple regions, then the portion of the database on which the query is operating should be distributed across the same regions to ensure local accesses. As the present data placement might not fit this, migrating pages from one NUMA region to another can be performed to improve the situation. To do so, different options exist: One option is to rely on automatic NUMA balancing integrated in Linux, which is steered by the observed access patterns and frequency. Another option is to actively trigger migration via the system call move_pages(). Unfortunately, both variants have significant downsides in terms of their feature set and performance. As an alternative, we propose a new user-space migration method called page_leap() that can perform page migration asynchronously at a high performance by exploiting features of the virtual memory subsystem. The method is (a) actively triggered by the user, (b) ensures that all pages are eventually migrated, (c) handles concurrent writes correctly, (d) supports pooled memory, (e) adaptively adjusts its migration granularity based on the workload, and (f) supports both small pages and huge pages.