Ascend-CC: Confidential Computing on Heterogeneous NPU for Emerging Generative AI Workloads

TL;DR

Ascend-CC is a novel confidential computing architecture leveraging heterogeneous NPUs to securely run large language models without trusting the host system, ensuring data and model protection with minimal performance overhead.

Contribution

It introduces a hardware-agnostic, device-centric confidential computing solution for heterogeneous NPUs that secures AI workloads without requiring modifications to existing AI software stacks.

Findings

Achieves strong security guarantees for LLMs on NPU hardware.

Implements delegation-based memory semantics for isolation.

Demonstrates minimal overhead in Llama2 and Llama3 evaluations.

Abstract

Cloud workloads have dominated generative AI based on large language models (LLM). Specialized hardware accelerators, such as GPUs, NPUs, and TPUs, play a key role in AI adoption due to their superior performance over general-purpose CPUs. The AI models and the data are often highly sensitive and come from mutually distrusting parties. Existing CPU-based TEEs such as Intel SGX or AMD SEV do not provide sufficient protection. Device-centric TEEs like Nvidia-CC only address tightly coupled CPU-GPU systems with a proprietary solution requiring TEE on the host CPU side. On the other hand, existing academic proposals are tailored toward specific CPU-TEE platforms. To address this gap, we propose Ascend-CC, a confidential computing architecture based on discrete NPU devices that requires no trust in the host system. Ascend-CC provides strong security by ensuring data and model encryption that protects not only the data but also the model parameters and operator binaries. Ascend-CC uses delegation-based memory semantics to ensure isolation from the host software stack, and task attestation provides strong model integrity guarantees. Our Ascend-CC implementation and evaluation with state-of-the-art LLMs such as Llama2 and Llama3 shows that Ascend-CC introduces minimal overhead with no changes in the AI software stack.