A2: Extracting Cyclic Switchings from DOB-nets for Rejecting Excessive Disturbances

TL;DR

This paper introduces an attention-based method to extract finite-state automata from DOB-nets trained with RL, revealing cyclic switching patterns that enhance understanding and robustness against external disturbances.

Contribution

It proposes the A² approach to extract a Key Moore Machine Network from DOB-nets, providing insights into their switching mechanisms under external disturbances.

Findings

Identified three cyclic switching patterns in DOB-net behavior.

Discovered control saturation synchronized with unknown disturbances.

Linked switching mechanisms to hybrid control design principles.

Abstract

Reinforcement Learning (RL) is limited in practice by its gray-box nature, which is responsible for insufficient trustiness from users, unsatisfied interpretation for human intervention, inadequate analysis for future improvement, etc. This paper seeks to partially characterize the interplay between dynamical environments and the DOB-net. The DOB-net obtained from RL solves a set of Partially Observable Markovian Decision Processes (POMDPs). The transition function of each POMDP is largely determined by the environments, which are excessive external disturbances in this research. This paper proposes an Attention-based Abstraction (A${}^2$) approach to extract a finite-state automaton, referred to as a Key Moore Machine Network (KMMN), to capture the switching mechanisms exhibited by the DOB-net in dealing with multiple such POMDPs. This approach first quantizes the controlled platform by learning continuous-discrete interfaces. Then it extracts the KMMN by finding the key hidden states and transitions that attract sufficient attention from the DOB-net. Within the resultant KMMN, this study found three patterns of cyclic switchings (between key hidden states), showing controls near their saturation are synchronized with unknown disturbances. Interestingly, the found switching mechanism has appeared previously in the design of hybrid control for often-saturated systems. It is further interpreted via an analogy to the discrete-event subsystem in the hybrid control.