OLAF: Programmable Data Plane Acceleration for Asynchronous Distributed Reinforcement Learning

TL;DR

This paper presents OLAF, a programmable data plane architecture that accelerates asynchronous distributed reinforcement learning by reducing model update staleness and network congestion through inline processing and novel queueing mechanisms.

Contribution

OLAF introduces a novel in-network processing architecture with a queueing mechanism and feedback control to mitigate staleness in distributed RL training.

Findings

Reduces model update staleness significantly.

Improves convergence rate in asynchronous DRL.

Maintains global fairness and responsiveness.

Abstract

Asynchronous Distributed Reinforcement Learning (DRL) can suffer from degraded convergence when model updates become stale, often the result of network congestion and packet loss during large-scale training. This work introduces a network data-plane acceleration architecture that mitigates such staleness by enabling inline processing of DRL model updates as they traverse the accelerator engine. To this end, we design and prototype a novel queueing mechanism that opportunistically combines compatible updates sharing a network element, reducing redundant traffic and preserving update utility. Complementing this we provide a lightweight transmission control mechanism at the worker nodes that is guided by feedback from the in-network accelerator. To assess model utility at line rate, we introduce the Age-of-Model (AoM) metric as a proxy for staleness and verify global fairness and responsiveness properties using a formal verification method. Our evaluations demonstrate that this architecture significantly reduces update staleness and congestion, ultimately improving the convergence rate in asynchronous DRL workloads.