Exploration Policies for On-the-Fly Controller Synthesis: A Reinforcement Learning Approach

TL;DR

This paper introduces a reinforcement learning-based heuristic for on-the-fly controller synthesis, enabling efficient, generalized strategy generation in non-deterministic environments without exhaustive exploration.

Contribution

It proposes a novel RL approach with a modified DQN to learn heuristics that generalize to larger problem instances, improving over domain-independent heuristics.

Findings

RL-based heuristics outperform existing heuristics in unseen instances

Heuristics learned on small problems generalize to larger instances

The approach enables zero-shot policy transfer in controller synthesis

Abstract

Controller synthesis is in essence a case of model-based planning for non-deterministic environments in which plans (actually ''strategies'') are meant to preserve system goals indefinitely. In the case of supervisory control environments are specified as the parallel composition of state machines and valid strategies are required to be ''non-blocking'' (i.e., always enabling the environment to reach certain marked states) in addition to safe (i.e., keep the system within a safe zone). Recently, On-the-fly Directed Controller Synthesis techniques were proposed to avoid the exploration of the entire -and exponentially large-environment space, at the cost of non-maximal permissiveness, to either find a strategy or conclude that there is none. The incremental exploration of the plant is currently guided by a domain-independent human-designed heuristic. In this work, we propose a new method for obtaining heuristics based on Reinforcement Learning (RL). The synthesis algorithm is thus framed as an RL task with an unbounded action space and a modified version of DQN is used. With a simple and general set of features that abstracts both states and actions, we show that it is possible to learn heuristics on small versions of a problem that generalize to the larger instances, effectively doing zero-shot policy transfer. Our agents learn from scratch in a highly partially observable RL task and outperform the existing heuristic overall, in instances unseen during training.