SpeedMalloc: Improving Multi-threaded Applications via a Lightweight Core for Memory Allocation

TL;DR

SpeedMalloc introduces a lightweight support-core for multi-threaded memory allocation, significantly reducing cache conflicts and synchronization overhead, leading to substantial performance improvements over existing allocators.

Contribution

It proposes a novel support-core architecture for memory allocators that enhances multi-threaded performance by minimizing cache conflicts and enabling flexible allocator design.

Findings

SpeedMalloc achieves up to 1.75x speedup over state-of-the-art allocators.

It reduces cache pollution and synchronization overhead in multi-threaded environments.

Performance gains are consistent across various multi-threaded workloads.

Abstract

Memory allocation, though constituting only a small portion of the executed code, can have a "butterfly effect" on overall program performance, leading to significant and far-reaching impacts. Despite accounting for just approximately 5% of total instructions, memory allocation can result in up to a 2.7x performance variation depending on the allocator used. This effect arises from the complexity of memory allocation in modern multi-threaded multi-core systems, where allocator metadata becomes intertwined with user data, leading to cache pollution or increased cross-thread synchronization overhead. Offloading memory allocators to accelerators, e.g., Mallacc and Memento, is a potential direction to improve the allocator performance and mitigate cache pollution. However, these accelerators currently have limited support for multi-threaded applications, and synchronization between cores and accelerators remains a significant challenge. We present SpeedMalloc, using a lightweight support-core to process memory allocation tasks in multi-threaded applications. The support-core is a lightweight programmable processor with efficient cross-core data synchronization and houses all allocator metadata in its own caches. This design minimizes cache conflicts with user data and eliminates the need for cross-core metadata synchronization. In addition, using a general-purpose core instead of domain-specific accelerators makes SpeedMalloc capable of adopting new allocator designs. We compare SpeedMalloc with state-of-the-art software and hardware allocators, including Jemalloc, TCMalloc, Mimalloc, Mallacc, and Memento. SpeedMalloc achieves 1.75x, 1.18x, 1.15x, 1.23x, and 1.18x speedups on multithreaded workloads over these five allocators, respectively.

Tables

Table 1. Table 1 . SOTA software and hardware memory allocators.

Work	Weakness/Feature	Multi-Thread
Jemalloc (Evans, 2006)	Cache pollution; Cross-core metadata synchronization overhead	Yes
TCMalloc (Google, 2024)		Yes
Mimalloc (Leijen et al., 2019)		Yes
Mallacc (Kanev et al., 2017)	Additional dedicated hardware; blowup issue in multi-threaded apps	No
Memento (Wang et al., 2023)		Limited
SpeedMalloc (Ours)	Additional lightweight co-processor	Yes

Table 2. Table 2 . gem5 simulator hardware configuration.

Main cores	Core	1-16 $\times$ AArch64 big core
	L1&L2 Cache	32KB 8-way L1d; 4 cycles latency
		32KB 8-way L1i; 4 cycles latency
		256KB 8-way L2; 12 cycles latency
	LLC	2MB per Core 16-way; 24 cycles latency
Support-Core	Core	1 $\times$ AArch64 little core
	HMQ	128-entry dispatch queue & response queue
	L1 Cache	16KB 4-way L1d; 2 cycles latency
		16KB 4-way L1i; 2 cycles latency
Main-Support Core Latency		8 cycles latency111111The low-latency can be achieved by bypassing the main core cache without triggering the cache line indexing, which has been applied in prior works (Wang et al., 2016; Wu et al., 2021) (more studies in § 6.4.2).
DRAM		16GB 2400MHz DDR4, 8 devices/rank, 2 ranks/channel, tCAS=tRCD=tRP=14.16ns, tRAS =32ns

Table 3. Table 3 . Percentage of instructions on memory allocation vs overall program speedups (over Jemalloc using 16 threads). SpeedMalloc improves the performance globally (more than just the allocator) due to the removal of metadata cache pollution and across-core synchronization.

Workload	Allocator Ins (%)	TCMalloc	Mimalloc	SpeedMalloc
Larson	$5.99\%$	$2.71\times$	$2.17\times$	$3.19\times$
Xmalloc	$2.45\%$	$1.06\times$	$1.09\times$	$1.16\times$
Scratch	$2.62\%$	$1.49\times$	$1.54\times$	$1.62\times$
Sh6bench	$5.55\%$	$1.63\times$	$1.45\times$	$1.73\times$
Sh8bench	$7.22\%$	$1.31\times$	$1.39\times$	$1.49\times$
Mstress	$5.46\%$	$1.65\times$	$1.62\times$	$1.71\times$
AllocTest	$3.91\%$	$1.04\times$	$1.40\times$	$1.46\times$
BFS	$3.07\%$	$2.55\times$	$2.50\times$	$3.57\times$
BC	$0.37\%$	$1.18\times$	$1.16\times$	$1.20\times$
DC	$6.94\%$	$1.10\times$	$1.39\times$	$1.64\times$
geo-mean	$4.36\%$	$1.48\times$	$1.52\times$	$1.75\times$

Full text

We present SpeedMalloc, using a lightweight support-core to process memory allocation tasks in multi-threaded applications. The support-core is a lightweight programmable processor with efficient cross-core data synchronization and houses all allocator metadata in its own caches. This design minimizes cache conflicts with user data and eliminates the need for cross-core metadata synchronization. In addition, using a general-purpose core instead of domain-specific accelerators makes SpeedMalloc capable of adopting new allocator designs. We compare SpeedMalloc with state-of-the-art software and hardware allocators, including Jemalloc, TCMalloc, Mimalloc, Mallacc, and Memento. SpeedMalloc achieves $1.75\times$, $1.18\times$, $1.15\times$, $1.23\times$, and $1.18\times$ speedups on multithreaded workloads over these five allocators, respectively.

††copyright: none

1. Introduction

Memory management and allocation consume a substantial proportion of datacenter computation cycles (Kanev et al., 2015; Sriraman and Dhanotia, 2020; Gonzalez et al., 2023; Zhou et al., 2024). Prior studies found that allocators impact the performance not only of allocation tasks but also the entire program (Li et al., 2023) and even the entire data center fleet (Hunter et al., 2021; Zhou et al., 2024). Prior works have shown that while only 2% of the code is spent on allocation, the overall execution time can vary up to 72% when using different allocators (Li et al., 2023) in a manner similar to butterfly effect (Lorenz, 1963) (where a small action somewhere can cause a large effect elsewhere). The effect is even stronger for multi-threaded applications. In our experiments with three industry-grade allocators, JeMalloc (from Meta), Mimalloc (from Microsoft), and TCMalloc (from Google), we observe up to $2.71\times$ overall application execution time variation in overall application execution time across the allocators, as illustrated in Figure 1(a) and (b) (both use 16 threads).

In modern multi-threaded multi-core systems, memory allocation is a complex task that needs to excel at many aspects, including (a) performing fast allocation and deallocation of objects, (b) handling objects of various sizes efficiently with minimal fragmentation, (c) returning freed space quickly for future allocations, (d) being scalable to many threads and cores, and so on. Efforts to achieve fast and concurrent memory allocation while minimizing fragmentation have been ongoing since the last century. Multiple software solutions, in the form of optimized malloc libraries, have been designed and implemented (Berger et al., 2002; Evans, 2006; Pheatt, 2008; Gloger, 2024; Google, 2024; Aigner et al., 2015; Leijen et al., 2019; Liétar et al., 2019; Hunter et al., 2021; Maas et al., 2021; Yang et al., 2023; Zhou et al., 2024). However, all modern software allocators face a common challenge: they rely on complex metadata structures. These sophisticated metadata structures negatively impact overall program performance in two key ways: cache pollution and metadata synchronization.

Allocator metadata is likely to pollute processor caches, by evicting cache lines containing program user data. Figure 1(c) shows profiling results for the BFS (Breadth-first Search) workload. We note that the conflicts with allocator metadata lead to $28.3\%$ of all cache misses in TCMalloc111We chose TCMalloc because of its wide-spread use in research and commercial datacenters, as well as its profiling utilities (Kanev et al., 2017; Hunter et al., 2021; Maas et al., 2021; Zhou et al., 2024). when using 16 threads. In addition to cache pollution, the cross-core metadata synchronization also limits the scalability of modern allocators. To support concurrent allocation in different threads, state-of-the-art allocators employ thread-local cache to maintain metadata associated with each thread (Evans, 2006; Google, 2024; Leijen et al., 2019). To avoid memory blowup(Berger et al., 2000) (memory consumption grows linearly with the number of threads), modern allocators use a centralized cross-thread free memory pool to balance the memory consumption of each thread (Leijen et al., 2019; Google, 2024; Hunter et al., 2021). Accessing both thread-local and cross-thread metadata is expensive. The logical thread and physical core mapping are transparent to allocators, resulting in non-negligible overhead for cross-core synchronization (Asgharzadeh et al., 2022). For example, in Larson (Larson and Krishnan, 1998) (a workload models realworld server-client behaviors) from mimalloc-bench (Microsoft, 2024), 22.8% of overall cycles are spent on cross-core synchronization (Figure 1(d)).

Prior solutions present an opportunity to use a hardware accelerator near the core for system functions (Karandikar et al., 2021, 2023; Wang et al., 2023; Kanev et al., 2017). However, memory allocation in multi-threaded scenarios has its own challenges, especially the coherency issue (more in § 2.3). Instead of “an accelerator per core”, we propose SpeedMalloc, a system that centralizes memory allocation tasks to a support-core. To reduce the cross-core synchronization overhead, SpeedMalloc employs a new communication interface that routes data directly to the support-core through the coherent network. We also add hardware support, i.e., hardware message queues, in the support-core to improve the allocation request throughput. By reducing the cache conflicts and alleviating the cross-core metadata synchronization issues, SpeedMalloc makes the CPU cycles more productive.

Another existing approach for reducing cache pollution and metadata synchronization (without triggering coherency issues) is to isolate the allocation functions from the rest of the application and offload the allocation tasks to a separate core in the system (Tiwari et al., 2010; Li et al., 2023). However, a full-fledged powerful core includes features that are not necessary for memory allocators (e.g., floating point arithmetic operations), resulting in energy inefficiency. Instead of a normal core, SpeedMalloc uses a lightweight support-core for memory allocation tasks. The lightweight support-core is a micro-controller grade core with about 1$$\sim$$2\% of a modern processor chip area (the area of an Intel Xeon Max socket is $400mm^{2}$, while the area of a simple RISC-V core can be less than $4mm^{2}$ (Wang et al., 2024)).

Results show that on multi-threaded workloads, SpeedMalloc achieves an average of $1.75\times$, $1.18\times$, and $1.15\times$ overall performance speedup over Jemalloc, TCMalloc, and Mimalloc, respectively. As a result of using a lightweight support-core, SpeedMalloc also achieves $1.69\times$, $1.15\times$, and $1.12\times$ energy savings over the three industry allocators. We also compare SpeedMalloc with two hardware allocator accelerators Mallacc (Kanev et al., 2017) and Memento (Wang et al., 2023). SpeedMalloc achieves $1.23\times$ and $1.18\times$ overall performance speedups over Mallacc and Memento on multi-threaded workloads, respectively.

2. Related Work & Motivation

This section includes an overview of memory allocators (§ 2.1) and their performance challenges with multi-threaded workloads (§ 2.2). We also describe the limitations of current hardware accelerators for multi-threaded applications (§ 2.3) and motivate the need for SpeedMalloc (§ 2.4).

2.1. Background of Memory Allocators

Memory allocation exists at two levels: user level and kernel level. A user-level memory allocator (UMA, referred to as memory allocator in this paper) implements user-level memory allocation tasks, such as ptmalloc2 (Gloger, 2024) (default Glibc allocator), TCMalloc (Hunter et al., 2021), and Mimalloc (Leijen et al., 2019). For kernel-level operations, UMAs use $mmap()$222In this paper, we will focus on the user-level memory management and use the default $mmap()$ system call. system calls to make operating systems map memory segments for each process.

We show the pseudo-code of $malloc()$ (UMA memory allocation interface) in Code 1. To minimize the overhead of $mmap()$ system calls, most UMAs request large chunks of memory at a time and allocate memory out of these chunks to satisfy allocation requests (known as the malloc_fast). UMAs only make additional $mmap()$ calls when pre-allocated space is exhausted (known as the malloc_generic). With the help of malloc_fast, state-of-the-art memory allocators spend less than 10% of the allocation cycles in the kernel space (Kanev et al., 2017; Zhou et al., 2024). Managing the pre-allocated memory in a way that strikes a balance between allocation speed and memory fragmentation remains a challenging problem, drawing research attention from both software and hardware communities.

2.2. Challenges in Multi-threaded UMAs

Modern UMAs support concurrent memory allocation and deallocation for multi-threaded applications. For example, TCMalloc (Google, 2024) assigns each thread/core a thread-local cache. The majority of allocations are satisfied from the thread-local cache on each core, instead of processed by a central allocator in series.

One major challenge for memory allocators in handling multi-threaded applications is memory blowup, i.e., memory consumption can grow linearly with the number of threads/cores (server-client model) (Berger et al., 2000). To address the blowup issue, state-of-the-art allocators (Leijen et al., 2019; Google, 2024) employ tiered metadata. As shown in Figure 2, allocators use thread-local cache (thread-local metadata) to support fast and small concurrent memory allocation in each thread. If the thread-local cache is exhausted, the allocation request will search for free memory space of other threads through the shared cache (inter-thread shared metadata).

Though such tiered metadata can help with the blowup issue, it makes the system suffer from (a) additional cache pollution, and (b) synchronization overhead.

Cache pollution: Accessing the metadata can pollute the CPU caches, by evicting cache lines containing user data (Li et al., 2023). In multi-threaded allocators, one allocation request can result in: (a) accessing the thread-local cache; (b) accessing the shared cache if the thread-local cache is exhausted, and potentially the thread-local cache of other threads for additional free space. Cache pollution poses an even greater challenge in multi-threaded applications since allocator metadata can even pollute cache lines of other cores (right part of Figure 2).

Synchronization: Modern allocators use intensive synchronization logic, e.g., atomic instructions (Asgharzadeh et al., 2022), to secure the metadata coherency (for both thread-local and shared cache). Cross-core synchronization can downgrade the system performance since the communication overhead increases with the number of cores in the system (Schweizer et al., 2015; Asgharzadeh et al., 2022).

2.3. Current Hardware Solutions are Limited

Offloading system functions (including memory allocators) to accelerators near the core have been proposed in the past (Rezaei and Kavi, 2006; Kanev et al., 2017; Karandikar et al., 2021, 2023; Wang et al., 2023). However, “an accelerator per core” solution falls short for multi-threaded systems. As shown in Figure 2 (right side), the tiered metadata of one thread can exist in multiple cores. Implementing a coherent protocol between all processors and all accelerators is expensive. An alternative solution is to make allocators not use shared cache and use thread-local cache only (Wang et al., 2023); however, this leaves the blowup issue unresolved.

Another limitation of accelerators is that accelerators are not flexible to software challenges. For example, TCMalloc has an average of $1.33$ new commits every day (Google, 2024). Regarding LOC (line of codes), last year, the most active author made $32.2$ LOC changes on average per day.

2.4. A Support-Core with Efficient Data Transfer

Instead of “an accelerator per core”, a centralized solution, using a support-core for all memory allocation tasks can help alleviate cache pollution and metadata coherency issues. Table 1 summarizes state-of-the-art software and hardware memory allocators. General-purpose333There are also other profiling and machine learning guided software allocators (Maas et al., 2020; Oh et al., 2021), which require offline profiling/training with additional runtime overhead. These methodologies are orthogonal to our approach in SpeedMalloc. software allocators (Evans, 2006; Google, 2024; Leijen et al., 2019) suffer from cache pollution and synchronization overhead. Current hardware allocators (Kanev et al., 2017; Wang et al., 2023) have limited support for multi-threaded applications.

While prior prototypes of offloading memory allocator to a separate core achieve performance gains for single-threaded applications (Tiwari et al., 2010; Li et al., 2023), their performance remains constrained by the substantial overhead of cross-core communication (more on this in § 6.4.2). For example, a single atomic instruction (required for inter-core synchronization) can consume up to 700 cycles in modern processors (Asgharzadeh et al., 2022), while most allocation functions can be finished within 100 cycles (Kanev et al., 2017; Zhou et al., 2024). A more desired and feasible direction is to offload memory allocators to a support-core with software and hardware support for efficient cross-core communication. In addition, compared with harvesting one of the primary processing cores, a lightweight support-core is adequate for memory allocation tasks, as they do not need floating-point and vector operations or complex control logic. Following this insight, we develop SpeedMalloc.

3. SpeedMalloc Overview

This section provides an overview of SpeedMalloc. As shown in Figure 3, we add a lightweight support-core connected to the coherent network. The support-core only needs an L1 cache to hold allocator metadata. The support-core does not require any floating point or vector units for allocation tasks. In the following sections, we elaborate on the design of SpeedMalloc. We develop a lightweight interface between main cores and the support-core for offloading memory allocators (§ 4). This reduces communication costs compared to offloading memory allocators to an idle core. We discuss the runtime of executing allocation tasks in the support-core (§ 5). SpeedMalloc uses both software and hardware approaches to reduce cache pollution and contention during allocation runtime.

4. SpeedMalloc Invocation Interface

In a conventional system, offloading a function to a separate core typically requires expensive cross-core synchronization schemes, e.g., atomic instructions like exclusive read/write. SpeedMalloc employs a lightweight function invocation interface by using signals and data packets (§ 4.1) with ISA and compiler support (§ 4.2 and § 4.3). Data packets can route to the support-core directly, without triggering additional cache snoop requests.

4.1. Signal and Data Packets

Figure 4 shows the allocation function (support-core) invocation interface in SpeedMalloc (using $malloc()$ as an example). The allocation function parameters (allocation request size) and process identification are transferred along with the signals. The right side of Figure 4 shows the data packet format for this purpose (more details about the data packet itself will be discussed in § 5.2).

The $malloc()$ process is broken down into 5 stages. Once the main core reaches the beginning of the $malloc()$ function, it will send the “start” signal to the support-core (Stage 1). Along with the “start” signal, the data packet is also transferred to the support-core. After receiving the “start” signal from the main cores, the support-core will start executing the $malloc()$ function (Stage 2). The main core that calls the support-core will skip the $malloc()$ execution. Meanwhile, the main core can execute instructions that do not depend on $malloc()$ return value. If no such instructions are available, the main core will enter the next stage (Stage 3) and wait for the “end” signal from the support-core. SpeedMalloc masks the exception and interrupt handler at this stage to guarantee the main core can receive the “end” signal. When the support-core reaches the end of the $malloc()$ function, it will send an “end” signal along with the returned allocated address to the main core to wake the core waiting for $malloc()$. At Stage 4, main cores can execute instructions that rely on the $malloc()$ return value. The support-core will update the allocator metadata asynchronously, removing this operation from the critical application execution path. Once the metadata is updated, the support-core will switch to Stage 5, awaiting the next “start” signal.

One prerequisite for the invocation interface is that signals must be aware of the program context information, e.g., program counter, of the allocation functions. Such information is architecture and compiler dependent, necessitating additional ISA (§ 4.2) and compiler (§ 4.3) support.

4.2. ISA Support

Main cores will send “start” signals to the support-core for each allocation request. Similarly, the support-core will send “end” signals back once finishing allocation. To support such signals, SpeedMalloc enhances current ISAs by adding 4 new unconditional branch instructions444Reusing existing instructions by utilizing reserved bits is also possible. We demonstrate using new instructions in this paper..

We define two new instructions for main cores, “mallocstart()” and “freestart()”. When executing these instructions, main cores will send the “start” signal to the support-core for both instructions. When committing these instructions, “mallocstart()” will check for the “end” signal and update the destination register value (return value of the $malloc()$ function). Since memory deallocation is not on the critical path, SpeedMalloc supports processing $free()$ functions asynchronously instead of blocking the execution of the main cores. Therefore, the “freestart()” instruction can retire without waiting for an end signal from the support-core.

We also define two new instructions for the support-core, “mallocend()” and “freeend()”. These two instructions are inserted at the end of malloc() and free() functions. These two instructions indicate the end of one allocation/deallocation function and will set the PC (program counter) to the start address of the following pending $malloc()$/$free()$ request if there are any. Otherwise, the support-core will stall the pipeline to save energy. The “mallocend()” will send the “end” signal and the destination register value to the main core, while “freeend()” retires as normal instructions.

4.3. Compiler Support

With the new instructions in SpeedMalloc, the main cores will have the information of when to branch to the support-core execution. However, where to branch is compiler-dependent. This part will discuss the compiler support for the added instructions in SpeedMalloc.

When the main cores issue “mallocstart()” (or “freestart()”) instructions, the support-core will start fetching instructions related to memory allocation tasks. Where to fetch the instructions in the support-core, i.e., the destination PC of the branch instructions, should be part of the instruction and generated during compilation time (instruction example shown in Figure 5). To solve this, we add a post-compilation pass555Instead of a post-compilation pass, this process can also be integrated into the current compiler, e.g., by modifying gcc. to get such information. We pack SpeedMalloc as either a static or shared library. Programmers can choose whether to use SpeedMalloc or other allocators during the program linking stage, without changing program source codes.

5. SpeedMalloc Runtime

In addition to the allocator invocation, SpeedMalloc provides additional support for allocation runtime. SpeedMalloc reduces the cache pollution and contention by using both software approaches (§ 5.1). SpeedMalloc handles contention well by additional hardware support in the support-core (§ 5.2).

5.1. Software Approach

SpeedMalloc employs a segregated metadata layout, making data accessed by the support-core separate from other data accessed by main cores, thus reducing metadata cache pollution (§ 5.1.1) and contention (§ 5.1.2).

5.1.1. Reduce Cache Pollution

During allocation runtime, accessing the same memory location from both main cores and support-core can result in significant performance overhead due to cache pollution. To avoid such behavior, SpeedMalloc employs a segregated metadata layout, making data accessed by the support-core separate from other data accessed by main cores.

As shown in Figure 6, SpeedMalloc maintains linked lists as metadata to indicate available free space with different sizes. During memory allocation, the support-core will only access the metadata and get the pointer of the next available memory block, i.e., the 64/128/256-byte66664/128/256 are block size examples. Design methodologies are well explored in software allocators (Leijen et al., 2019; Google, 2024) and are not the focus of SpeedMalloc. block in the right part of Figure 6. During the allocation phase, the support-core can make $mmap()$ system calls (if necessary) to claim memory from the OS, making the majority of system memory metadata accessed exclusively by the support-core. After the support-core finishes allocation tasks, the main cores will receive the address pointer for the allocated memory. Main cores will only access the allocated memory blocks, which are segregated from allocator metadata, which resolves the cache pollution issue in other software allocators.

5.1.2. Address Metadata Contention

In the previous discussion of the invocation interface in SpeedMalloc, assuming only one main core interacts with the support-core. However, metadata contention occurs when multiple cores simultaneously attempt to access allocator metadata (e.g., shared cache), which is a common issue in memory allocation for multi-threaded applications. Software solutions use atomic instructions to serialize changes made to the allocator metadata (Figure 2) to avoid race conditions; however, the overhead of these atomic instructions is high.

With the segregated metadata layout in SpeedMalloc, all allocation requests are processed by the support-core, eliminating such metadata contention. All allocator metadata is physically stored within the support-core, preventing inter-core metadata exchanges. This design eliminates the need for atomic instructions, unlike other software allocators. In addition, such a design in SpeedMalloc can also eliminate the blowup issue and reduce the design complexity of the allocator itself.

5.2. Hardware Approach

SpeedMalloc eliminates metadata contention that exists in modern allocators but comes at the cost of hardware contention. Hardware contention occurs when multiple cores send allocation requests to the support-core simultaneously: the support-core has to process these requests serially.

SpeedMalloc relieves the hardware contention by using additional hardware message queues in the support-core (HMQ shown in Figure 7). When the support-core receives a memory allocation request from main cores, the dispatcher first pushes the request to the $malloc()$ queue or $free()$ queue777When the $malloc()$/$free()$ queue is full, additional requests could be buffered using reserved physical memory (Margaritov et al., 2021) and fetched to the queue once space becomes available., depending on the type of the request. The scheduler will pop the first element from the $malloc()$ queue if it is not empty, or the first element from the $free()$ queue if the $malloc()$ queue is empty. The scheduler prioritizes $malloc()$ requests since it is on the (performance) critical path. In addition, the scheduler serves allocation requests from different cores in a round-robin fashion (main core ID is part of the data packet shown in Figure 4), which gives each main core fair access to the support-core. After finishing the execution of $malloc()$/$free()$ on the support-core, the returned results will be pushed to the response queue (if it has). The support-core will then pop the first element from the response queue and send the “end” signal and the return value (if it has) back to the main core.

In addition to the two message queues, we add one register buffer in the support-core. The register buffer is implemented as a directly mapped cache. It uses the process ID as the index and stores required system control and segment registers for address translation. Instead of transferring these system registers in each data packet (Figure 4) on the fly, the main core will only send them to the support-core in its first allocation request (transferring data packets can be implemented as writing to the device memory). This also helps achieve a faster allocation invocation in SpeedMalloc.

6. Evaluation

We demonstrate SpeedMalloc ’s effectiveness by answering the following questions through experimental evaluation:

• •

Can SpeedMalloc improve overall program performance when compared with other allocators? If so, what are the sources of the performance gains? (§ 6.2)

• •

Can SpeedMalloc achieve energy savings given the added support-core in the system? (§ 6.3)

• •

Is SpeedMalloc necessary? Can the same performance gains be achieved using existing hardware, e.g., partitioning cache or harvesting an idle core? (§ 6.4)

• •

What is the scalability of SpeedMalloc? (§ 6.5)

Highlights of our results are:

• •

SpeedMalloc achieves $1.75\times$, $1.18\times$, $1.15\times$, $1.23\times$, and $1.18\times$ speedups over Jemalloc, TCMalloc, Mimalloc, Mallacc, and Memento+ when using 16 threads. The gains are from the reduction of cache pollution and the elimination of metadata synchronization.

• •

SpeedMalloc achieves $1.69\times$, $1.15\times$, $1.12\times$, $1.26\times$, and $1.22\times$ energy savings over Jemalloc, TCMalloc, Mimalloc, Mallacc, and Memento+ on the 16-core system (running 16 threads). The reason is that SpeedMalloc uses a lightweight support-core, which consumes only \sim$$2\% of the system power.

6.1. Experimental Setup

6.1.1. End-to-End Simulation Platforms

We evaluate SpeedMalloc using gem5 (Binkert et al., 2011; Lowe-Power et al., 2020) in full system mode888We are not able to prototype SpeedMalloc in a real system, e.g., offloading to an FPGA, as the support-core has to be connected to the coherent bus above the last-level-cache level to achieve a low communication overhead, which requires additional ISA and hardware not exist in current systems.. The detailed configuration of the hardware simulated is shown in Table 2. We use the OoO (Out-of-Order) processor model for the big core and the minor (in-order) processor model for the little core (gem, 2024). We use GCC 9.4 as the compiler with -O3 optimization to compile SpeedMalloc and all workloads. We also use the “-march=armv8.1-a” optimization to support far atomic instruction in recent aarch64 processors. We use McPAT (Li et al., 2009) to obtain area, power, and energy values.

6.1.2. Benchmarks

We evaluate SpeedMalloc on both single-threaded and multi-threaded workloads. The workloads are from Mimalloc-bench (Microsoft, 2024) (Mimalloc’s evaluate suite (Leijen et al., 2019)999Jemalloc’s and TCMalloc’s benchmark suites are not publicly available.), GAPBS (Beamer et al., 2015) (graph processing), Redis-benchmark (red, 2024) (in-memory database), and NAS (Bailey et al., 1991) (computational fluid dynamics applications). We simulated the **end-to-end101010Simulating a subset using simpoints (Sherwood et al., 2002) would not work since SpeedMalloc needs to analyze how allocator impacts on the whole program performance. This makes it impractical to simulate large workloads, such as SPEC benchmarks, which are also unsuitable for memory allocation studies (Zhou et al., 2024). In addition, most of the SPEC benchmarks do not actively allocate objects in a stable state, which makes them unsuitable for studying memory allocation (Zhou et al., 2024). ** performance of each workload, with instruction counts from 47 million to 7 billion.

Single-threaded workloads. We use two single-threaded workloads Espresso (Grunwald et al., 1993) and Cfrac (Barrett, 2024) from Mimalloc-bench. Espresso (Grunwald et al., 1993) is a programmable logic array analyzer, in the context of cache-aware memory allocation. Cfrac (Barrett, 2024) is an implementation of continued fraction factorization, using many small short-lived allocations. We use 6 workloads from Redis-benchmark (red, 2024): LPUSH, RPUSH, LPOP, RPOP, SADD, and SPOP.

Multi-threaded workloads. We selected 7 representative multi-threaded workloads from Mimalloc-bench, each exhibiting a different allocation characteristic and showing significant performance differences when using different allocators (Leijen et al., 2019). Larson (Larson and Krishnan, 1998; Dang et al., 2022) simulates realworld server-client behaviors. Xmalloc (Boreham, 2000) simulates the scenario where objects allocated by one thread are freed by another thread. Cache-Scratch (Berger et al., 2000) (abbreviated as Scratch) is used to test passive-false sharing of cache lines. Sh6(8)bench (MicroQuill, 2024) is a stress test for allocating small objects more frequently. Mstress (Leijen, 2024) simulates real-world server-like allocation patterns, where objects can migrate between threads with lifetimes. Alloctest (AG, 2018) simulates intensive allocation workloads with a Pareto size distribution.

We selected 2 widely-used graph algorithms from GAPBS: Breadth First Search (BFS) and Betweenness Centrality (BC), using a synthetic graph with 64k nodes with an average degree of 100. These two workloads use queue data structures that would allocate/deallocate the heap memory frequently, while other workloads, e.g., DFS (Depth First Search) based algorithms, use stack memory through recursive function calls (stack memory grows automatically instead of being managed by memory allocators).

We selected Data Cube (Frumkin and Shabano, 2003) (DC) from NAS, which benchmarks grid capabilities to handle the data movement across large distributed data sets. DC is also the only workload that allocates heap memory during runtime in the NAS suite.

Workloads are representative. Table. 3 shows the percentage of allocation instruction of the overall program for the benchmarks. For the 10 multi-threaded workloads we used, on average $4.4\%$ of instructions is spent on memory allocation tasks. In addition, there is a large variance (from $0.37\%$ to $7.22\%$) of the overall instructions on memory allocation. Such variance indicates these workloads are highly representative and are able to cover both allocation intensive and non-intensive workloads. The suite we select is a good proxy for realworld workloads, which spans the full gamut of memory allocation overheads (from $2.7\%$ to $10.7\%$ and an average of \sim$$5\% of overall execution time is spent on memory allocation in Google datacenter workloads (Hunter et al., 2021)).

6.1.3. Baselines

We compare SpeedMalloc with three state-of-the-art software solutions and two hardware solutions.

Software baselines. Our software baselines are Jemalloc from Meta (Evans, 2006), TCMalloc from Google (Google, 2024), and Mimalloc from Microsoft (Leijen et al., 2019). For a fair comparison, we use the default hyperparameter (of each allocator) for all workloads. These state-of-the-art UMAs are chosen as the baselines for comparison as they are highly optimized for industry-scale warehouse systems.

Hardware baselines. There are other hardware accelerators for single-threaded memory allocation, e.g., Mallacc (Kanev et al., 2017) and Memento (Wang et al., 2023). Mallacc was built atop of TCMalloc121212It is infeasible to build Mallacc atop Mimalloc, which uses aggregated metadata layout (address space of metadata and user data are intertwined) (Li et al., 2023). and the key idea of Mallacc is to have a dedicated malloc cache at the L1-cache level. We implemented Mallacc by adding a 16KB malloc cache. Memento achieved further performance improvement by implementing an accelerator near the core (named Object Allocator) dedicated to allocation tasks. To support multi-threaded systems and avoid memory blowup (details in § 2.3) in the original Memento design, we implemented Memento+ by adopting the shared cache (transfer cache) design of TCMalloc for cross-thread communication and connecting both the Object Allocator and malloc cache to the coherent bus. We set the Object Allocator latency in Memento+ the same as Memento (each allocation request takes the same time as a single cache access if it is a malloc cache hit (Wang et al., 2023)).

6.2. Does SpeedMalloc achieve performance gains?

6.2.1. Results

SpeedMalloc performs well on single-threaded workloads. As shown in Figure 8, SpeedMalloc achieves an average of $1.09\times$, $1.08\times$, $1.04\times$, and $1.03\times$ speedups over Jemalloc, TCMalloc, Mimalloc, and Mallacc. SpeedMalloc outperforms the three software allocators by reducing cache pollution. When compared with Jemalloc, TCMalloc, and Mimalloc, SpeedMalloc achieves $19.93\%$, $14.69\%$, and $13.65\%$ reduction in L2 miss cycles.

Memento+ uses a dedicated accelerator for fast allocation (allocation cycles contribute \sim$$3\% of overall cycles for the 8 single-threaded workloads) but only buffers 16 entries (based on serverless object life cycles) of each allocation size. The allocation size of Redis-benchmark is more unified, making SpeedMalloc perform better due to adopting general software allocators (e.g., Mimalloc can buffer at most 8k entries for the 8-byte allocation size). In general, SpeedMalloc achieves similar performance as Memento+131313Memento+ outperforms Jemalloc by \sim$$1.1\times in Redis-benchmark. This is consistent with Memento results (Wang et al., 2023)..

For single-threaded applications, Memento+ addressed the metadata cache pollution issue well since there is no cross-thread metadata sharing and synchronization overhead. For multi-threaded workloads, where shared metadata pollutes the cache and synchronization overhead occurs, SpeedMalloc is more effective.

SpeedMalloc performs much better on multi-threaded workloads. As shown in Figure 9, SpeedMalloc outperforms the three state-of-the-art software allocators for all multi-threaded workloads. On average, SpeedMalloc achieves $1.39\times$, $1.40\times$, $1.58\times$, $1.73\times$, and $1.75\times$ speedups over Jemalloc for 1, 2, 4, 8, and 16 threads, respectively. Even when compared with the most recent software allocator TCMalloc, SpeedMalloc achieves $1.21\times$, $1.15\times$, $1.18\times$, $1.16\times$, and $1.18\times$ speedups for 1, 2, 4, 8, and 16 threads, respectively. Compared to Mimalloc, SpeedMalloc achieves average speedups of $1.17\times$, $1.22\times$, $1.25\times$, $1.21\times$, and $1.15\times$ for 1, 2, 4, 8, and 16 threads. Compared with two hardware accelerators Mallacc and Memento+, SpeedMalloc also achieves $1.23\times$ and $1.18\times$ speedups when using 16 threads.

Magnitude of the Butterfly Effect: Table 3 shows the percentage of allocation instructions of the overall program for the benchmarks. For the 10 multi-threaded workloads we used, on average $4.4\%$ of the overall instructions is spent on memory allocation tasks. However, much like the butterfly effect, even though only $4.4\%$ of total instructions is spent on memory allocation tasks, this can ultimately lead to $1.75\times$ increase in overall program performance. In addition, there is a large variance (from $0.4\%$ to $7.2\%$) of the overall instructions spent on memory allocation. Such variance indicates that the chosen workloads are covering a large workload space encompassing allocation-intensive and non-intensive workloads. SpeedMalloc achieves performance improvement in all scenarios.

6.2.2. Analysis

We analyze the sources of the performance gains. We break down the execution time to analyze the performance gains from reduced cache pollution and eliminating metadata synchronization.

SpeedMalloc reduces cache pollution. Figure 10 shows the L2 miss cycles141414 We cannot get the cache line evictions due to metadata access for allocators using aggregated metadata layout (Mimalloc). Instead, we use the L2 miss cycles as a global metric to indicate reducing cache miss results in better allocator performance. on the 16-core system (running 16 threads). Since all allocator metadata are stored in the support-core, SpeedMalloc achieves $42.36\%$, $18.76\%$, and $22.80\%$ L2 miss cycles reduction compared to Jemalloc, TCMalloc, and Mimalloc, respectively. The L2 miss has a non-negligible impact on the program performance, which contributes $24.22\%$, $17.18\%$, and $18.08\%$ of the overall execution cycles of the 10 workloads (GEO-MEAN) when using Jemalloc, TCMalloc, and Mimalloc. Considering the speedups for the entire program, $10.26\%$ (out of $42.93\%$) of the SpeedMalloc execution time reduction over Jemalloc is due to the reduction of L2 misses. Likewise, $4.78\%$ (out of $15.30\%$) and $6.28\%$ (out of $13.06\%$) of execution time reduction over TCMalloc and Mimalloc come from L2 miss reduction. Mallacc and Memento+ used distributed malloc cache and accelerators for memory allocation, which did not address the shared cache metadata pollution in multi-threaded scenarios. As a result, the $3.73\%$ (out of $15.02\%$) performance improvement of SpeedMalloc over Memento+ comes from the L2 miss reduction.

SpeedMalloc eliminates metadata synchronization. SpeedMalloc eliminates metadata synchronization incurred in multi-threaded software allocators by removing atomic instructions. We quantify the overhead of these atomic instructions in other allocators (shown in Figure 11). SpeedMalloc reduces the atomic instruction cycles for all workloads. Considering the average of the 10 multi-threaded workloads, $3.57\%$ (out of $42.93\%$), $11.99\%$ (out of $15.30\%$), $11.73\%$ (out of $13.06\%$), $10.34\%$ (out of $18.71\%$), and $10.81\%$ (out of $15.02\%$) of the SpeedMalloc execution time reduction over Jemalloc, TCMalloc, Mimalloc, Mallacc, and Memento+ is due to the removal of cross-core metadata synchronization.

Other factors affecting performance. The average performance gains (of the 10 multi-threaded workloads) of SpeedMalloc over TCMalloc and Mimalloc are almost consistent with the two individual factors (e.g., for Mimalloc $13.06\%$ $\approx$ $6.28\%+11.73\%$). There are also other potential factors affecting allocator performance. SpeedMalloc also benefits from these aspects, which results in more speedups over Jemalloc.

First, SpeedMalloc achieves a reduction in the number of instructions over Jemalloc. We built SpeedMalloc atop TCMalloc and Mimalloc. On average of the 10 workloads, TCMalloc and Mimalloc saves 11.1% and 13.9% instructions compared with Jemalloc, respectively. The number of instructions saved is even larger than the time spent on memory allocation. The reason is that TCMalloc and Mimalloc handle address mapping well and help reduce the number of page faults (which are handled out of the allocation phase in the kernel space). In addition, with the removal of metadata contention, SpeedMalloc uses shared metadata (to indicate available free spaces) for all threads. This makes SpeedMalloc achieve an additional 4.97% instruction savings over TCMalloc. Cache line false sharing is another reason, which indicates multiple small objects are allocated within the same cache line but shared by multiple cores. Mimalloc and TCMalloc handle cache line false sharing better than Jemalloc (Leijen et al., 2019) (Scratch exemplifies this aspect).

Memory allocation is sophisticated and can affect the program execution from many perspectives. In addition to performance gains from reducing cache pollution and eliminating metadata synchronization, SpeedMalloc can also benefit from these other factors mentioned above.

Memory Consumption. We evaluate the memory consumption of SpeedMalloc. SpeedMalloc adopts state-of-the-art software allocator design and addresses blowup well. SpeedMalloc can achieve more memory savings due to the use of centralized metadata, which reduces memory fragmentation in thread-local data. However, SpeedMalloc may also trigger additional memory consumption since the support-core priorities allocation requests, delaying recycling memory from deallocation requests, which increases peak memory consumption (Figure 7). Overall (shown in Figure 12), SpeedMalloc consumes nearly the same (\sim$$1\% difference) memory compared with TCMalloc and Mimalloc (Mallacc and Memento+ are built atop TCMalloc, we assume they have the same memory consumption).

6.3. Does SpeedMalloc achieve energy savings?

SpeedMalloc also achieves energy savings, although the added support-core consumes additional power. Compared with a regular out-of-order core, the support-core is a simple in-order core without any floating point or vector units (and registers), instruction renaming units, re-order buffers, etc. As a result, the power of a support-core is estimated to be $33.72\%$ that of a main core, and the area of a support-core is only $24.43\%$ that of a main core (results obtained using McPAT (Li et al., 2009)). Considering a 16-core system (with memory controllers), the support-core only consumes $1.50\%$ and $2.25\%$ of the entire process chip area and power, respectively.

Figure 13 shows that SpeedMalloc achieves energy savings over the three software allocators when using 2 or more main cores. On average of the 10 multi-threaded workloads, SpeedMalloc achieves $1.20\times$, $1.44\times$, $1.63\times$, and $1.69\times$ energy savings over Jemalloc for 2, 4, 8, and 16 main cores; $0.98\times$, $1.08\times$, $1.11\times$, and $1.15\times$ compared to TCMalloc; $1.03\times$, $1.13\times$, $1.15\times$, and $1.12\times$ compared to Mimalloc. Comparing against the two hardware accelerator solutions Mallacc and Memento+, SpeedMalloc achieves $1.26\times$ and $1.22\times$ savings, as these hardware solutions require additional hardware resources for each core. Therefore, using a support-core for memory allocation is feasible since the power overhead is negligible.

We also perform sensitivity studies on the support-core microarchitecture configures. Figure 14 shows the execution time, power, and energy of running workload Sh6bench using 16 main cores (with one support-core). By increasing the L1-dcache capacity from 1KB to 16KB, there is a $1.27\times$ speedup while only $2.1\%$ additional power consumption. Overall, 16KB L1-dcache capacity is the most energy-efficient design (SpeedMalloc also has the flexibility of using a 4KB L1-dcache in power-constrained embedded systems).

6.4. SpeedMalloc vs. Using Existing Hardware

SpeedMalloc achieves better performance than state-of-the-art software memory allocators, by introducing additional architectural support over the current system. Other potential solutions may not require hardware changes: using cache partitions (§ 6.4.1) or harvesting an idle core in the system (§ 6.4.2). However, these solutions have their limitations and cannot achieve the efficiency of SpeedMalloc.

6.4.1. SpeedMalloc vs. Cache partition

Cache pollution may be eliminated by partitioning caches and allocating dedicated ways (assuming a set-associative cache) to allocator metadata. However, dedicating ways for allocator metadata may adversely impact some applications since less amount of cache is available for application user data. On the other hand, SpeedMalloc is a more workload-agnostic solution. We compare SpeedMalloc with cache partition in this section.

We partition the 8-way L2 into two groups, one dedicated to allocator metadata and the other for all other program data, e.g., in Figure 15, 7-1 indicates that 1 way is exclusive for allocator metadata. As the results suggest, the cache partition does not secure a performance gain on all applications and results in a 7\%$$\sim$$12\% slowdown.

The partition schema is workload-dependent, thus, using a static partition will not achieve performance gains for all workloads. Dynamic cache partition has been used in prior works (Sanchez and Kozyrakis, 2011; Mittal, 2017), but still faces many challenges. First, the overhead of additional control logic for tracking cache usage to determine proper partition is non-negligible (Sanchez and Kozyrakis, 2011; Mittal, 2017). Additionally, whether a $malloc$ request invokes $mmap$ system calls is unpredictable, which results in significant memory access difference (Google, 2024; Mirosoft, 2024). These challenges prevent cache partition from becoming a viable alternative to SpeedMalloc. SpeedMalloc is a more general solution and improves the performance for all workloads, not only because it addresses cache pollution, but also multi-threaded metadata synchronization (§ 6.2.2).

Possibility of increasing cache capacity. Instead of cache partition, increasing cache capacity may reduce cache pollution. We show the performance of Mimalloc using different cache capacities in Figure 16 (256KB L2 is the baseline in Table 2). On average for the 10 workloads, increasing the cache capacity by $2\times$ (from 128KB to 256KB) can result in $1.04\times$ speedup and increasing cache size by $8\times$ (from 128KB to 1MB) can result in $1.17\times$ speedup.

We find that increasing cache capacity can lead to performance improvements in workloads that are bounded by cache misses. For example, the performance Mimalloc on LarsonN is more bounded by cache misses (Figure 10) instead of atomic cycles (Figure 11), so increasing L2 cache capacity can also bring performance improvement. However, increasing the cache capacity for workloads that are bounded by cache misses, e.g., AllocTest, could be wasteful. Moreover, increasing cache capacity (either makes the additional capacity exclusively for memory allocators or not): (a) cannot resolve the metadata synchronization issue (§ 6.2.2), (b) may help with metadata capacity miss but not with conflict miss, (c) cannot simplify allocator design (§ 6.2.2), and (d) adds to cache access latency (7zi, 2024), area, and power overheads.

6.4.2. SpeedMalloc vs. Harvesting an idle core

Instead of introducing an additional support-core along with hardware enhancements, one may wonder if it is possible to reduce cache pollution and metadata synchronization by offloading allocation tasks to one of the idle cores. In this section, we compare the performance of SpeedMalloc with using an idle core (we refer to this as IC-Malloc) for memory allocation. We use the same gem5 setup for a fair comparison.

Figure 17 shows the performance comparison between TCMalloc, IC-Malloc, and SpeedMalloc when using 16 threads. Although IC-Malloc effectively reduces cache conflicts, its scalability is constrained by existing cross-core communication interfaces. IC-Malloc cannot outperform TCMalloc considering the average of the 10 workloads. In addition, using a full-fledged core for memory allocation tasks is unnecessary and leads to higher energy consumption.

SpeedMalloc outperforms IC-Malloc with the invocation interface and runtime support. The signal system (§ 4) in SpeedMalloc is necessary for a fast allocation invocation. The hardware message queues (§ 5.2) in the support-core, which helps buffering and scheduling allocation requests in allocation-intensive workloads, is missing in IC-Malloc. Without these supports, the cross-core communication in IC-Malloc can only be implemented using atomic instructions with the current cache coherency model. We break down the performance benefits of these supports in Figure 17. Decoupling the allocator from the main process (§ 5.1) cannot make IC-Malloc achieve better performance than TCMalloc. With the signal system for fast invocation (§ 4), a $1.09\times$ speedup over TCMalloc can be achieved. Finally, with the hardware message queues to relieve contention (§ 5.2), SpeedMalloc can achieve a $1.18\times$ speedup over TCMalloc.

6.5. Scalability of SpeedMalloc

Our analysis (§ 6.2) shows that one support-core can serve memory allocation requests from 16 main cores. In modern many-core workstation systems, e.g., Apple’s M3 Max chip which has 12 performance cores and 4 efficiency cores. We suggest using one of the efficiency cores as the support core. We also discuss how SpeedMalloc scales for larger warehouse-scale computers in this section.

To scale for warehouse-scale computers (64+ cores), SpeedMalloc can employ multiple support cores. For example, contemporary systems such as AMD’s Zen architecture (Naffziger et al., 2021; Bhargava and Troester, 2024) have eight cores per chiplet and Intel’s recent Xeon architecture has 15 cores per chiplet (Nassif et al., 2022); one can consider adding one support-core per core complex. This setup is suitable for container-based datacenter hyperscale workloads (Stojkovic et al., 2023; Sahraei et al., 2023), where cross-chiplet data transfer is minimal. For monolithic architecture, SpeedMalloc could connect all support-cores in a manner similar to the connection of last-level cache slices in scalable coherency fabric, such as in the Grace super chip (Elster and Haugdahl, 2022; Nvidia, 2024). SpeedMalloc could also use a more powerful support-core for memory allocation-intensive workloads. This remains an energy-efficient solution, as the power budget of the support-core increases with the number of main cores. Additionally, SpeedMalloc could optimize the allocator software design by using a local cache for every 16 cores to reduce cross-support-core metadata synchronization.

7. Conclusion

Memory allocator performance is an important issue as demonstrated by many recent works (Hunter et al., 2021; Zhou et al., 2024). Many efforts have been reported to improve memory allocators over the past few decades, but few have addressed the interference from the allocator metadata in multi-threaded applications.

In this paper, we presented SpeedMalloc by using a lightweight support-core for memory allocation tasks. With SpeedMalloc the overall program performance benefits from reducing cache pollution by not bringing metadata into main core caches and removing unnecessary inter-core metadata synchronization. We compared SpeedMalloc with Meta’s Jemalloc, Google’s TCMalloc, Microsoft’s Mimalloc, and two accelerators Mallacc and Memento+. With a full system simulation of the entire program, we have shown that SpeedMalloc achieves $1.75\times$ speedup and $1.69\times$ energy savings over Jemalloc. When compared to TCMalloc and Mimalloc, SpeedMalloc achieves $1.18\times$ and $1.15\times$ speedups, and $1.15\times$ and $1.12\times$ energy savings. Compared with accelerators, SpeedMalloc also achieves $1.23\times$ and $1.18\times$ speedups over state-of-the-art accelerators Mallacc and Memento+.

The minimal hardware modifications in the support-core make it programmable and capable of offloading additional system functions. Since memory allocation is a fundamental component in all applications, the cumulative effect of multiple optimizations, each improving performance by a few percent, can result in substantial CPU cycle savings, particularly when deployed at scale across megawatt datacenters.