A Loss Landscape Visualization Framework for Interpreting Reinforcement Learning: An ADHDP Case Study

TL;DR

This paper introduces a comprehensive visualization framework for understanding reinforcement learning dynamics, demonstrated through an ADHDP case study in spacecraft attitude control, revealing how different components influence learning stability.

Contribution

The authors extend a critic match loss landscape visualization into a multi-perspective framework that clarifies the interactions of value estimation, policy optimization, and TD signals during training.

Findings

Visualization reveals how training stabilizers affect the loss landscape.

Comparison of ADHDP variants shows landscape changes impact learning stability.

Framework provides systematic insights into RL behavior across different algorithms.

Abstract

Reinforcement learning algorithms have been widely used in dynamic and control systems. However, interpreting their internal learning behavior remains a challenge. In the authors' previous work, a critic match loss landscape visualization method was proposed to study critic training. This study extends that method into a framework which provides a multi-perspective view of the learning dynamics, clarifying how value estimation, policy optimization, and temporal-difference (TD) signals interact during training. The proposed framework includes four complementary components; a three-dimensional reconstruction of the critic match loss surface that shows how TD targets shape the optimization geometry; an actor loss landscape under a frozen critic that reveals how the policy exploits that geometry; a trajectory combining time, Bellman error, and policy weights that indicates how updates move across the surface; and a state-TD map that identifies the state regions that drive those updates. The Action-Dependent Heuristic Dynamic Programming (ADHDP) algorithm for spacecraft attitude control is used as a case study. The framework is applied to compare several ADHDP variants and shows how training stabilizers and target updates change the optimization landscape and affect learning stability. Therefore, the proposed framework provides a systematic and interpretable tool for analyzing reinforcement learning behavior across algorithmic designs.