Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO

TL;DR

This paper investigates the vulnerabilities of decentralized Group Relative Policy Optimization (GRPO) in training large language models, demonstrating effective adversarial attacks and proposing defenses, including logit-based filtering and LLM judging.

Contribution

It introduces the first adversarial attacks on decentralized GRPO and proposes two novel defense mechanisms to improve robustness.

Findings

Adversaries can achieve up to 100% attack success rate in 50 iterations.

Proposed defenses effectively prevent most attacks, except DoS.

Code for attacks and defenses is publicly available at https://github.com/gensyn-ai/HTTT.

Abstract

Group Relative Policy Optimization (GRPO) has demonstrated wide adoption in the post-training of Large Language Models (LLMs). In GRPO, prompts are answered by the model and preferred behaviour is learnt via reinforcement learning. Owing to the small communication volume, GRPO is inherently suitable for decentralised training as the prompts can be concurrently answered by multiple nodes and these completions are exchanged in the form of strings. In this work, we explore the robustness of decentralised GRPO by presenting the first adversarial attacks and countermeasures. We present a diverse set of attacks where malicious nodes poison benign models by sharing their poisoned completions. We demonstrate these attacks on math and coding tasks and show that an adversary can achieve attack success rates of up to 100% in as few as 50 iterations. Moreover, to mitigate the attacks, we propose two defense mechanisms that check logit probabilities of completions or utilize an LLM judge to filter completions. The defenses prevent all but the DoS attack that causes unnecessarily lengthy but conceptually correct completions. The code of both attacks and defenses can be found at: https://github.com/gensyn-ai/HTTT.