A Safe Exploration Strategy for Model-free Task Adaptation in Safety-constrained Grid Environments

TL;DR

This paper presents a novel safe exploration framework for model-free reinforcement learning agents in safety-constrained grid environments, enabling safer adaptation to new tasks with fewer safety violations.

Contribution

The authors introduce a pre-training and classification-based approach to identify unsafe states, guiding agents to follow safe policies and improve safety during exploration.

Findings

Reduced safety violations during adaptation to new environments

Effective identification of unsafe states using trained classifiers

Agents successfully learn optimal policies with safety constraints

Abstract

Training a model-free reinforcement learning agent requires allowing the agent to sufficiently explore the environment to search for an optimal policy. In safety-constrained environments, utilizing unsupervised exploration or a non-optimal policy may lead the agent to undesirable states, resulting in outcomes that are potentially costly or hazardous for both the agent and the environment. In this paper, we introduce a new exploration framework for navigating the grid environments that enables model-free agents to interact with the environment while adhering to safety constraints. Our framework includes a pre-training phase, during which the agent learns to identify potentially unsafe states based on both observable features and specified safety constraints in the environment. Subsequently, a binary classification model is trained to predict those unsafe states in new environments that exhibit similar dynamics. This trained classifier empowers model-free agents to determine situations in which employing random exploration or a suboptimal policy may pose safety risks, in which case our framework prompts the agent to follow a predefined safe policy to mitigate the potential for hazardous consequences. We evaluated our framework on three randomly generated grid environments and demonstrated how model-free agents can safely adapt to new tasks and learn optimal policies for new environments. Our results indicate that by defining an appropriate safe policy and utilizing a well-trained model to detect unsafe states, our framework enables a model-free agent to adapt to new tasks and environments with significantly fewer safety violations.