Safe Reinforcement Learning Beyond Baseline Control: A Hierarchical Framework for Space Triangle Tethered Formation System

TL;DR

This paper introduces a hierarchical reinforcement learning framework combining model-based control and SAC to improve precision, efficiency, and safety in space tethered formations, validated through simulations.

Contribution

A novel hierarchical RL control framework for TTFS that integrates baseline control with SAC, enhancing deployment accuracy and energy efficiency.

Findings

Reduces steady-state tether tracking errors by over 96%.

Decreases fuel consumption by two orders of magnitude.

Proves closed-loop stability using Lyapunov methods.

Abstract

Triangular tethered formation system (TTFS) provide a promising platform for deep space exploration and distributed sensing due to its intrinsic spatial-orientation stability and capability of adjusting distances among node satellites through deployment and retrieval of tethers. However, due to the coupled tether-satellite dynamics and disturbance sensitivity of TTFS, traditional control methods struggle to achieve a balanced trade-off among configuration accuracy requirements, tension constraints, and energy efficiency consumption throughout the deployment process.In this paper, a novel model-reference reinforcement learning control framework is proposed for TTFS. By integrating baseline model-based control with a Soft Actor-Critic (SAC) compensator, the proposed method simultaneously achieves high-precision tracking, fuel efficiency, and compliance with tension limits. A hierarchical training scheme is developed to address the convergence difficulties arising from strongly coupled states in centralized training, while tailored reward functions, reset conditions, and normalization criteria are designed to accelerate training convergence. Closed-loop stability of the overall control law is rigorously proven using Lyapunov methods. Simulation results demonstrate that the proposed controller reduces steady-state tracking errors by over 96% for tethers and 99% for node satellites, while cutting fuel consumption by two orders of magnitude compared with the baseline method. These results validate the effectiveness and stability of the proposed approach for TTFS deployment control.