ProToken: Token-Level Attribution for Federated Large Language Models

TL;DR

ProToken introduces a method for token-level attribution in federated LLMs, enabling identification of client contributions during text generation while preserving privacy, with high accuracy across multiple models and domains.

Contribution

It presents ProToken, a novel provenance methodology that achieves accurate client attribution at the token level in federated LLMs without compromising privacy.

Findings

Achieves 98% average attribution accuracy.

Maintains high accuracy with increasing number of clients.

Effective across diverse LLM architectures and domains.

Abstract

Federated Learning (FL) enables collaborative training of Large Language Models (LLMs) across distributed data sources while preserving privacy. However, when federated LLMs are deployed in critical applications, it remains unclear which client(s) contributed to specific generated responses, hindering debugging, malicious client identification, fair reward allocation, and trust verification. We present ProToken, a novel Provenance methodology for Token-level attribution in federated LLMs that addresses client attribution during autoregressive text generation while maintaining FL privacy constraints. ProToken leverages two key insights to enable provenance at each token: (1) transformer architectures concentrate task-specific signals in later blocks, enabling strategic layer selection for computational tractability, and (2) gradient-based relevance weighting filters out irrelevant neural activations, focusing attribution on neurons that directly influence token generation. We evaluate ProToken across 16 configurations spanning four LLM architectures (Gemma, Llama, Qwen, SmolLM) and four domains (medical, financial, mathematical, coding). ProToken achieves 98% average attribution accuracy in correctly localizing responsible client(s), and maintains high accuracy when the number of clients are scaled, validating its practical viability for real-world deployment settings.