Gradient Inversion in Federated Reinforcement Learning

TL;DR

This paper introduces RGIA, a novel attack method that reconstructs private data in federated reinforcement learning by enforcing prior knowledge of environment dynamics, revealing privacy vulnerabilities.

Contribution

The paper proposes RGIA, a new gradient inversion attack that incorporates transition dynamics regularization to improve data reconstruction in FRL.

Findings

RGIA effectively reconstructs local data in control and autonomous driving tasks.

Regularization narrows solution space to true transition dynamics.

Reconstructed data closely matches real environment transitions.

Abstract

Federated reinforcement learning (FRL) enables distributed learning of optimal policies while preserving local data privacy through gradient sharing.However, FRL faces the risk of data privacy leaks, where attackers exploit shared gradients to reconstruct local training data.Compared to traditional supervised federated learning, successful reconstruction in FRL requires the generated data not only to match the shared gradients but also to align with real transition dynamics of the environment (i.e., aligning with the real data transition distribution).To address this issue, we propose a novel attack method called Regularization Gradient Inversion Attack (RGIA), which enforces prior-knowledge-based regularization on states, rewards, and transition dynamics during the optimization process to ensure that the reconstructed data remain close to the true transition distribution.Theoretically, we prove that the prior-knowledge-based regularization term narrows the solution space from a broad set containing spurious solutions to a constrained subset that satisfies both gradient matching and true transition dynamics.Extensive experiments on control tasks and autonomous driving tasks demonstrate that RGIA can effectively constrain reconstructed data transition distributions and thus successfully reconstruct local private data.