ECLIP: Energy-efficient and Practical Co-Location of ML Inference on Spatially Partitioned GPUs

TL;DR

ECLIP is a framework that enhances energy efficiency and throughput of co-located ML inference on GPUs by minimizing repartitioning overheads through kernel-wise resource partitioning and optimized CU assignment.

Contribution

ECLIP introduces a low-overhead, kernel-wise resource partitioning framework with a resource optimizer to improve GPU utilization and energy efficiency during ML inference co-location.

Findings

13% throughput improvement

25% energy efficiency gain

Reduced repartitioning overheads

Abstract

As AI inference becomes mainstream, research has begun to focus on improving the energy consumption of inference servers. Inference kernels commonly underutilize a GPU's compute resources and waste power from idling components. To improve utilization and energy efficiency, multiple models can co-locate and share the GPU. However, typical GPU spatial partitioning techniques often experience significant overheads when reconfiguring spatial partitions, which can waste additional energy through repartitioning overheads or non-optimal partition configurations. In this paper, we present ECLIP, a framework to enable low-overhead energy-efficient kernel-wise resource partitioning between co-located inference kernels. ECLIP minimizes repartitioning overheads by pre-allocating pools of CU masked streams and assigns optimal CU assignments to groups of kernels through our resource allocation optimizer. Overall, ECLIP achieves an average of 13% improvement to throughput and 25% improvement to energy efficiency.