Investigating the Impact of Choice on Deep Reinforcement Learning for Space Controls

TL;DR

This paper examines how the number of discrete choices in reinforcement learning affects the performance of space control tasks, finding that limited choices excel in inspection tasks while continuous control is better for docking.

Contribution

It provides an analysis of discrete versus continuous action spaces in RL for space control tasks, highlighting task-dependent optimal control strategies.

Findings

Limited discrete choices optimize inspection task performance.

Continuous control yields better results for docking tasks.

Task-specific control strategies improve space autonomous operations.

Abstract

For many space applications, traditional control methods are often used during operation. However, as the number of space assets continues to grow, autonomous operation can enable rapid development of control methods for different space related tasks. One method of developing autonomous control is Reinforcement Learning (RL), which has become increasingly popular after demonstrating promising performance and success across many complex tasks. While it is common for RL agents to learn bounded continuous control values, this may not be realistic or practical for many space tasks that traditionally prefer an on/off approach for control. This paper analyzes using discrete action spaces, where the agent must choose from a predefined list of actions. The experiments explore how the number of choices provided to the agents affects their measured performance during and after training. This analysis is conducted for an inspection task, where the agent must circumnavigate an object to inspect points on its surface, and a docking task, where the agent must move into proximity of another spacecraft and "dock" with a low relative speed. A common objective of both tasks, and most space tasks in general, is to minimize fuel usage, which motivates the agent to regularly choose an action that uses no fuel. Our results show that a limited number of discrete choices leads to optimal performance for the inspection task, while continuous control leads to optimal performance for the docking task.