Theodosian: A Deep Dive into Memory-Hierarchy-Centric FHE Acceleration

TL;DR

This paper analyzes the memory bottlenecks in GPU-accelerated FHE, revealing the memory wall and low kernel utilization, and introduces Theodosian, a set of optimizations that significantly improve performance and reduce latency.

Contribution

It provides a detailed microarchitectural analysis of FHE on GPUs and proposes Theodosian, a novel memory-aware optimization framework that enhances FHE performance.

Findings

Memory wall persists beyond off-chip DRAM bottleneck.

Low intra-kernel parallelism limits throughput.

Theodosian improves CKKS performance by up to 1.83x.

Abstract

Fully homomorphic encryption (FHE) enables secure computation on encrypted data, mitigating privacy concerns in cloud and edge environments. However, due to its high compute and memory demands, extensive acceleration research has been pursued across diverse hardware platforms, especially GPUs. In this paper, we perform a microarchitectural analysis of CKKS, a popular FHE scheme, on modern GPUs. Focusing on the memory hierarchy, we demonstrate that dominant kernels remain bound by the on-chip L2 cache despite its high bandwidth, exposing a persistent inner memory wall beyond the conventional off-chip DRAM bottleneck. Further, we reveal that the overall CKKS throughput is constrained by low per-kernel hardware utilization, caused by insufficient intra-kernel parallelism. Motivated by these findings, we introduce Theodosian, a set of complementary, memory-aware optimizations that improve cache efficiency and reduce runtime overheads. Theodosian achieves 1.45--1.83x performance improvements over a highly optimized baseline, Cheddar, across representative CKKS workloads. On an RTX 5090, we reduce the bootstrapping latency for 32,768 complex numbers from 22.1ms to 15.2ms, and further to 12.8ms with additional algorithmic optimizations, establishing a new state-of-the-art GPU performance to the best of our knowledge.