Cascade: Token-Sharded Private LLM Inference

TL;DR

Cascade introduces a scalable, token-sharded inference protocol for large language models that balances privacy with performance, enabling practical secure third-party LLM deployment.

Contribution

It proposes a novel token-sharded inference scheme that improves scalability and speed over existing cryptographic privacy methods for large language models.

Findings

Cascade is resistant to advanced privacy attacks.

It significantly outperforms existing secure inference schemes in speed.

The method enables practical privacy-preserving inference for large LLMs.

Abstract

As LLMs continue to increase in parameter size, the computational resources required to run them are available to fewer parties. Therefore, third-party inference services -- where LLMs are hosted by third parties with significant computational resources -- are becoming increasingly popular. However, third party inference raises critical concerns about user data privacy. To mitigate these risks, privacy researchers have developed provably secure schemes for third-party inference, such as Secure Multi-Party Computation (SMPC). However, SMPC protocols have significant computational and communication overhead, and do not scale to large models. In this work, we propose a new multi-party inference protocol, Cascade, that avoids these punitive costs by leveraging sharding in the sequence dimension to maintain privacy, trading off cryptographic privacy guarantees for increased performance and scalability. We demonstrate that Cascade is resistant to a generalization of a recent attack that is highly effective against other statistical privacy schemes, and that it is further resistant to learning-based attacks. As Cascade is orders of magnitude faster than existing schemes, our findings offer practical solutions for secure deployment of modern state-of-the-art LLMs.