A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentiality

TL;DR

This paper introduces SOLID, a framework for on-premises LLM deployment that balances privacy and model confidentiality by securing bottom layers, defending against distillation attacks while maintaining customization capabilities.

Contribution

The paper reveals that securing bottom layers offers stronger protection against attacks and proposes SOLID, a method to optimize the number of secured layers for privacy and customization balance.

Findings

Securing bottom layers provides better protection than top layers.

SOLID outperforms baselines in balancing security and customization.

Extensive experiments on models from 1.3B to 70B parameters validate effectiveness.

Abstract

Privacy-sensitive users require deploying large language models (LLMs) within their own infrastructure (on-premises) to safeguard private data and enable customization. However, vulnerabilities in local environments can lead to unauthorized access and potential model theft. To address this, prior research on small models has explored securing only the output layer within hardware-secured devices to balance model confidentiality and customization. Yet this approach fails to protect LLMs effectively. In this paper, we discover that (1) query-based distillation attacks targeting the secured top layer can produce a functionally equivalent replica of the victim model; (2) securing the same number of layers, bottom layers before a transition layer provide stronger protection against distillation attacks than top layers, with comparable effects on customization performance; and (3) the number of secured layers creates a trade-off between protection and customization flexibility. Based on these insights, we propose SOLID, a novel deployment framework that secures a few bottom layers in a secure environment and introduces an efficient metric to optimize the trade-off by determining the ideal number of hidden layers. Extensive experiments on five models (1.3B to 70B parameters) demonstrate that SOLID outperforms baselines, achieving a better balance between protection and downstream customization.