Co-Activation Graph Analysis of Safety-Verified and Explainable Deep Reinforcement Learning Policies

TL;DR

This paper introduces a novel approach combining RL policy model checking with co-activation graph analysis to interpret and verify safe decision-making in deep reinforcement learning policies.

Contribution

It presents a new method that integrates model checking and co-activation graphs to analyze and interpret safe RL policies, enhancing understanding of neural network decision processes.

Findings

Effective interpretation of RL policies through co-activation graphs

Identification of unsafe behaviors in RL policies

Applicability demonstrated in various experimental settings

Abstract

Deep reinforcement learning (RL) policies can demonstrate unsafe behaviors and are challenging to interpret. To address these challenges, we combine RL policy model checking--a technique for determining whether RL policies exhibit unsafe behaviors--with co-activation graph analysis--a method that maps neural network inner workings by analyzing neuron activation patterns--to gain insight into the safe RL policy's sequential decision-making. This combination lets us interpret the RL policy's inner workings for safe decision-making. We demonstrate its applicability in various experiments.