Importance of Environment Design in Reinforcement Learning: A Study of a Robotic Environment

TL;DR

This paper emphasizes the importance of environment design in reinforcement learning by analyzing a robotic environment modeled as an MDP, and presents an exact solution approach using Bellman equations to understand optimal policies.

Contribution

It introduces a method to compute exact optimal policies in RL environments using Bellman equations solved with Wolfram Mathematica, avoiding approximation methods.

Findings

Exact optimal policies can be derived for RL environments.

Small modifications in environment schema lead to different optimal policies.

Provides insights into action selection mechanisms in RL.

Abstract

An in-depth understanding of the particular environment is crucial in reinforcement learning (RL). To address this challenge, the decision-making process of a mobile collaborative robotic assistant modeled by the Markov decision process (MDP) framework is studied in this paper. The optimal state-action combinations of the MDP are calculated with the non-linear Bellman optimality equations. This system of equations can be solved with relative ease by the computational power of Wolfram Mathematica, where the obtained optimal action-values point to the optimal policy. Unlike other RL algorithms, this methodology does not approximate the optimal behavior, it gives the exact, explicit solution, which provides a strong foundation for our study. With this, we offer new insights into understanding the action selection mechanisms in RL by presenting various small modifications on the very same schema that lead to different optimal policies.