On the impact of MDP design for Reinforcement Learning agents in Resource Management

TL;DR

This paper empirically analyzes how different Markov Decision Process (MDP) designs affect Reinforcement Learning agents' performance in resource management, highlighting the importance of state representation and transferability.

Contribution

It compares four MDP variations, demonstrating that compact state representations enable effective transfer of agents across environments with minimal retraining.

Findings

Compact state representations improve transferability.

Transferred agents outperform specialized ones in 80% of scenarios.

MDP design choices significantly impact agent performance.

Abstract

The recent progress in Reinforcement Learning applications to Resource Management presents MDPs without a deeper analysis of the impacts of design decisions on agent performance. In this paper, we compare and contrast four different MDP variations, discussing their computational requirements and impacts on agent performance by means of an empirical analysis. We conclude by showing that, in our experiments, when using Multi-Layer Perceptrons as approximation function, a compact state representation allows transfer of agents between environments, and that transferred agents have good performance and outperform specialized agents in 80\% of the tested scenarios, even without retraining.