NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs

TL;DR

NestedFP introduces a memory-efficient method for supporting dual-precision FP16 and FP8 inference in large language models, enabling dynamic SLO management with minimal quality and performance trade-offs.

Contribution

It proposes a novel overlay technique that allows FP8 parameters to share memory with FP16 parameters, reducing memory overhead for dual-precision LLM serving.

Findings

Supports both FP16 and FP8 with minimal quality loss

Reduces memory overhead by overlaying FP8 onto FP16 parameters

Maintains high inference throughput across modes

Abstract

Meeting service-level objectives (SLOs) in Large Language Models (LLMs) serving is critical, but managing the high variability in load presents a significant challenge. Recent advancements in FP8 inference, backed by native hardware support, offer a potential solution: executing FP16 models by default, while switching to FP8 models during sudden load surges to achieve higher throughput at the cost of a slight quality degradation. Although this approach facilitates effective SLO management, it introduces additional memory overhead due to storing two versions of the same model. In response, this paper proposes NestedFP, an LLM serving technique that supports both FP16 and FP8 models in a memory-efficient manner by overlaying FP8 parameters onto FP16 parameters, allowing both models to share the same FP16 memory footprint. By leveraging a compact data format for the overlay and a specialized GEMM kernel optimized for this format, NestedFP ensures minimal degradation in both model quality and inference throughput across both FP8 and FP16 modes. NestedFP provides a flexible platform for dynamic, SLO-aware precision selection. The code is available at https://github.com/SNU-ARC/NestedFP.