Particle Swarm Optimization for Generating Interpretable Fuzzy Reinforcement Learning Policies

TL;DR

This paper introduces FPSRL, a novel method combining particle swarm optimization with fuzzy reinforcement learning, trained on world models derived from previous data, to generate interpretable policies without online learning.

Contribution

It is the first to connect self-organizing fuzzy controllers with model-based batch RL, enabling safe, offline policy training in systems with known dynamics.

Findings

High-performing fuzzy policies achieved on benchmark tasks

Effective in domains where online learning is unsafe or impractical

Demonstrates interpretability and efficiency of the proposed approach

Abstract

Fuzzy controllers are efficient and interpretable system controllers for continuous state and action spaces. To date, such controllers have been constructed manually or trained automatically either using expert-generated problem-specific cost functions or incorporating detailed knowledge about the optimal control strategy. Both requirements for automatic training processes are not found in most real-world reinforcement learning (RL) problems. In such applications, online learning is often prohibited for safety reasons because online learning requires exploration of the problem's dynamics during policy training. We introduce a fuzzy particle swarm reinforcement learning (FPSRL) approach that can construct fuzzy RL policies solely by training parameters on world models that simulate real system dynamics. These world models are created by employing an autonomous machine learning technique that uses previously generated transition samples of a real system. To the best of our knowledge, this approach is the first to relate self-organizing fuzzy controllers to model-based batch RL. Therefore, FPSRL is intended to solve problems in domains where online learning is prohibited, system dynamics are relatively easy to model from previously generated default policy transition samples, and it is expected that a relatively easily interpretable control policy exists. The efficiency of the proposed approach with problems from such domains is demonstrated using three standard RL benchmarks, i.e., mountain car, cart-pole balancing, and cart-pole swing-up. Our experimental results demonstrate high-performing, interpretable fuzzy policies.