Multi-Phase Spacecraft Trajectory Optimization via Transformer-Based Reinforcement Learning

TL;DR

This paper presents a transformer-based reinforcement learning framework that unifies multi-phase spacecraft trajectory optimization, enabling adaptive, coherent control across different mission stages without manual phase transitions.

Contribution

The work introduces a novel transformer-augmented RL architecture that replaces traditional recurrent networks, allowing a single policy to handle multiple mission phases seamlessly.

Findings

Achieves near-optimal performance on simple benchmarks.

Successfully extends to complex multiphase rocket ascent tasks.

Maintains control stability and coherence across diverse mission regimes.

Abstract

Autonomous spacecraft control for mission phases such as launch, ascent, stage separation, and orbit insertion remains a critical challenge due to the need for adaptive policies that generalize across dynamically distinct regimes. While reinforcement learning (RL) has shown promise in individual astrodynamics tasks, existing approaches often require separate policies for distinct mission phases, limiting adaptability and increasing operational complexity. This work introduces a transformer-based RL framework that unifies multi-phase trajectory optimization through a single policy architecture, leveraging the transformer's inherent capacity to model extended temporal contexts. Building on proximal policy optimization (PPO), our framework replaces conventional recurrent networks with a transformer encoder-decoder structure, enabling the agent to maintain coherent memory across mission phases spanning seconds to minutes during critical operations. By integrating a Gated Transformer-XL (GTrXL) architecture, the framework eliminates manual phase transitions while maintaining stability in control decisions. We validate our approach progressively: first demonstrating near-optimal performance on single-phase benchmarks (double integrator and Van der Pol oscillator), then extending to multiphase waypoint navigation variants, and finally tackling a complex multiphase rocket ascent problem that includes atmospheric flight, stage separation, and vacuum operations. Results demonstrate that the transformer-based framework not only matches analytical solutions in simple cases but also effectively learns coherent control policies across dynamically distinct regimes, establishing a foundation for scalable autonomous mission planning that reduces reliance on phase-specific controllers while maintaining compatibility with safety-critical verification protocols.