Data-Driven Evaluation of Training Action Space for Reinforcement Learning

TL;DR

This paper introduces a Shapley-inspired, data-driven methodology for selecting and ranking training action spaces in reinforcement learning, significantly reducing search complexity and improving model efficiency.

Contribution

It presents a novel Monte Carlo-based approach for categorizing and ranking training actions, applicable across various domains and RL algorithms.

Findings

Reduces search space by 80%

Effectively categorizes actions into dispensable and indispensable

Facilitates high-performance, cost-efficient RL model design

Abstract

Training action space selection for reinforcement learning (RL) is conflict-prone due to complex state-action relationships. To address this challenge, this paper proposes a Shapley-inspired methodology for training action space categorization and ranking. To reduce exponential-time shapley computations, the methodology includes a Monte Carlo simulation to avoid unnecessary explorations. The effectiveness of the methodology is illustrated using a cloud infrastructure resource tuning case study. It reduces the search space by 80\% and categorizes the training action sets into dispensable and indispensable groups. Additionally, it ranks different training actions to facilitate high-performance yet cost-efficient RL model design. The proposed data-driven methodology is extensible to different domains, use cases, and reinforcement learning algorithms.