Privacy-Preserving LLMs Routing

TL;DR

PPRoute is a framework that enables privacy-preserving large language model routing using cryptographic techniques, achieving high performance and low latency while maintaining routing quality.

Contribution

It introduces novel MPC-friendly operations, a multi-step training algorithm, and an efficient Top-k algorithm for secure sorting in LLM routing.

Findings

Achieves plaintext-level routing performance under MPC.

Provides approximately 20× speedup over naive MPC implementations.

Reduces communication latency with an $O(1)$ complexity Top-k algorithm.

Abstract

Large language model (LLM) routing has emerged as a critical strategy to balance model performance and cost-efficiency by dynamically selecting services from various model providers. However, LLM routing adds an intermediate layer between users and LLMs, creating new privacy risks to user data. These privacy risks have not been systematically studied. Although cryptographic techniques such as Secure Multi-Party Computation (MPC) enable privacy-preserving computation, their protocol design and implementation remain under-explored, and na\"ive implementations typically incur prohibitive computational overhead. To address this, we propose a privacy-preserving LLM routing framework (PPRoute). PPRoute includes multiple strategies to speed up encoder inference and nearest neighbor search under the MPC and maintain the quality of LLM routing. First, PPRoute uses MPC-friendly operations to boost the encoder inference. Second, PPRoute uses a multiple-step model training algorithm to maintain routing quality despite the constraints of the encrypted domain. Third, PPRoute proposes an unsorted Top-k algorithm with $O(1)$ communication complexity for secure sorting in model search, significantly reducing communication latency. Across different datasets, PPRoute achieves the performance of plaintext counterparts, while achieving approximately a 20$\times$ speedup over na\"ive MPC implementations.