Separable Expert Architecture: Toward Privacy-Preserving LLM Personalization via Composable Adapters and Deletable User Proxies

TL;DR

This paper introduces a novel architecture that separates user data from shared model weights, enabling privacy-preserving personalization and deterministic unlearning through composable adapters and user proxies.

Contribution

It proposes a three-layer architecture that isolates personal data, allowing for efficient unlearning and enhanced privacy in large language models.

Findings

Per-user data influences outputs but remains isolated from shared weights.

Proxy removal returns the model to baseline, confirming effective unlearning.

Architecture mitigates risks like model inversion and membership inference.

Abstract

Current model training approaches incorporate user information directly into shared weights, making individual data removal computationally infeasible without retraining. This paper presents a three-layer architecture that decouples personal data from shared weights by combining a static base model, composable domain-expert LoRA adapters that shape behavior without imparting user data, and per-user proxy artefacts whose deletion constitutes deterministic unlearning. Evaluation on Phi-3.5-mini and Llama-3.1-8B confirms per-user differentiation in which personal data influences outputs while remaining isolated, verified by a return to baseline after proxy removal (KL divergence of approximately 0.21 nats, 82-89% verification pass rate) and near-zero cross-user contamination. Because user-specific information never enters shared weights, the architecture mitigates model inversion, membership inference, and training-data extraction against shared model components by construction. The approach converts machine unlearning from an intractable weight-editing problem into a deterministic deletion operation that preserves personalization alongside privacy-enhancing guarantees and is compatible with differentially private stochastic gradient descent (DP-SGD) for privacy-preserving shared model improvement.