Settling the Communication Complexity for Distributed Offline Reinforcement Learning

TL;DR

This paper establishes fundamental limits on communication efficiency in distributed offline reinforcement learning, providing lower bounds and algorithms that achieve near-optimal risk under strict communication constraints.

Contribution

It introduces the first minimax lower bounds for distributed offline RL and proposes algorithms that nearly attain these bounds under single-round communication.

Findings

Lower bounds on communication bits scale as Ω(AC) for contextual bandits.

Proposed algorithms based on least-squares and Monte-Carlo estimates achieve near-optimal risk.

Temporal difference methods are less effective under the communication constraints.

Abstract

We study a novel setting in offline reinforcement learning (RL) where a number of distributed machines jointly cooperate to solve the problem but only one single round of communication is allowed and there is a budget constraint on the total number of information (in terms of bits) that each machine can send out. For value function prediction in contextual bandits, and both episodic and non-episodic MDPs, we establish information-theoretic lower bounds on the minimax risk for distributed statistical estimators; this reveals the minimum amount of communication required by any offline RL algorithms. Specifically, for contextual bandits, we show that the number of bits must scale at least as $\Omega(AC)$ to match the centralised minimax optimal rate, where $A$ is the number of actions and $C$ is the context dimension; meanwhile, we reach similar results in the MDP settings. Furthermore, we develop learning algorithms based on least-squares estimates and Monte-Carlo return estimates and provide a sharp analysis showing that they can achieve optimal risk up to logarithmic factors. Additionally, we also show that temporal difference is unable to efficiently utilise information from all available devices under the single-round communication setting due to the initial bias of this method. To our best knowledge, this paper presents the first minimax lower bounds for distributed offline RL problems.