A Reference Architecture of Reinforcement Learning Frameworks

TL;DR

This paper proposes a comprehensive reference architecture for reinforcement learning frameworks by analyzing 18 existing systems, identifying common components, and outlining trends to facilitate comparison, evaluation, and improvement.

Contribution

It introduces the first reference architecture for RL frameworks based on empirical analysis of multiple implementations, aiding standardization and development.

Findings

Identified recurring architectural components in RL frameworks

Reconstructed characteristic RL patterns using the RA

Outlined architectural trends and improvement paths

Abstract

The surge in reinforcement learning (RL) applications gave rise to diverse supporting technology, such as RL frameworks. However, the architectural patterns of these frameworks are inconsistent across implementations and there exists no reference architecture (RA) to form a common basis of comparison, evaluation, and integration. To address this gap, we propose an RA of RL frameworks. Through a grounded theory approach, we analyze 18 state-of-the-practice RL frameworks and, by that, we identify recurring architectural components and their relationships, and codify them in an RA. To demonstrate our RA, we reconstruct characteristic RL patterns. Finally, we identify architectural trends, e.g., commonly used components, and outline paths to improving RL frameworks.