Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption

TL;DR

This paper introduces a privacy-preserving method for text classification that encrypts BERT embeddings using homomorphic encryption, preventing leakage of sensitive information while maintaining classification accuracy.

Contribution

The authors develop a homomorphic encryption-based privatization mechanism for BERT embeddings, enabling secure text classification without exposing private data.

Findings

Effective encryption of BERT embeddings with minimal utility loss

Supports GPU-accelerated CKKS encryption scheme

Maintains high classification accuracy on encrypted data

Abstract

Embeddings, which compress information in raw text into semantics-preserving low-dimensional vectors, have been widely adopted for their efficacy. However, recent research has shown that embeddings can potentially leak private information about sensitive attributes of the text, and in some cases, can be inverted to recover the original input text. To address these growing privacy challenges, we propose a privatization mechanism for embeddings based on homomorphic encryption, to prevent potential leakage of any piece of information in the process of text classification. In particular, our method performs text classification on the encryption of embeddings from state-of-the-art models like BERT, supported by an efficient GPU implementation of CKKS encryption scheme. We show that our method offers encrypted protection of BERT embeddings, while largely preserving their utility on downstream text classification tasks.