Personalized Language Model Learning on Text Data Without User Identifiers

TL;DR

This paper introduces a method for personalized language models that maintains user privacy by using dynamic, user-specific distributions for embeddings, enabling personalization without user identifiers.

Contribution

It proposes a novel approach where each device maintains a user-specific distribution for embeddings, enhancing privacy while improving language model personalization.

Findings

Significant accuracy improvements with anonymous user embeddings

Effective privacy preservation through distribution-based embeddings

Maintains real-time inference performance

Abstract

In many practical natural language applications, user data are highly sensitive, requiring anonymous uploads of text data from mobile devices to the cloud without user identifiers. However, the absence of user identifiers restricts the ability of cloud-based language models to provide personalized services, which are essential for catering to diverse user needs. The trivial method of replacing an explicit user identifier with a static user embedding as model input still compromises data anonymization. In this work, we propose to let each mobile device maintain a user-specific distribution to dynamically generate user embeddings, thereby breaking the one-to-one mapping between an embedding and a specific user. We further theoretically demonstrate that to prevent the cloud from tracking users via uploaded embeddings, the local distributions of different users should either be derived from a linearly dependent space to avoid identifiability or be close to each other to prevent accurate attribution. Evaluation on both public and industrial datasets using different language models reveals a remarkable improvement in accuracy from incorporating anonymous user embeddings, while preserving real-time inference requirement.