Investigating the Impact of Observation Space Design Choices On Training Reinforcement Learning Solutions for Spacecraft Problems

TL;DR

This paper explores how different observation space design choices, including sensor inclusion and reference frame orientation, affect the training efficiency and performance of reinforcement learning agents in spacecraft inspection tasks.

Contribution

It systematically investigates the impact of sensor design and reference frame choices on RL training outcomes for spacecraft control tasks.

Findings

Sensors generally improve learning efficiency and policy quality.

Reference frame consistency has a minor but positive effect.

Sensors are not strictly necessary for successful learning.

Abstract

Recent research using Reinforcement Learning (RL) to learn autonomous control for spacecraft operations has shown great success. However, a recent study showed their performance could be improved by changing the action space, i.e. control outputs, used in the learning environment. This has opened the door for finding more improvements through further changes to the environment. The work in this paper focuses on how changes to the environment's observation space can impact the training and performance of RL agents learning the spacecraft inspection task. The studies are split into two groups. The first looks at the impact of sensors that were designed to help agents learn the task. The second looks at the impact of reference frames, reorienting the agent to see the world from a different perspective. The results show the sensors are not necessary, but most of them help agents learn more optimal behavior, and that the reference frame does not have a large impact, but is best kept consistent.