Safe Exploration for Interactive Machine Learning

TL;DR

This paper introduces a framework that enhances existing unsafe interactive machine learning algorithms with safety guarantees by efficiently learning safety constraints using Gaussian process priors, reducing unnecessary exploration.

Contribution

It presents a novel add-on method that makes any unsafe IML algorithm safe by exploiting regularity assumptions to learn about safety efficiently.

Findings

Outperforms existing algorithms empirically

Efficiently learns safety constraints with Gaussian processes

Reduces unnecessary unsafe exploration

Abstract

In Interactive Machine Learning (IML), we iteratively make decisions and obtain noisy observations of an unknown function. While IML methods, e.g., Bayesian optimization and active learning, have been successful in applications, on real-world systems they must provably avoid unsafe decisions. To this end, safe IML algorithms must carefully learn about a priori unknown constraints without making unsafe decisions. Existing algorithms for this problem learn about the safety of all decisions to ensure convergence. This is sample-inefficient, as it explores decisions that are not relevant for the original IML objective. In this paper, we introduce a novel framework that renders any existing unsafe IML algorithm safe. Our method works as an add-on that takes suggested decisions as input and exploits regularity assumptions in terms of a Gaussian process prior in order to efficiently learn about their safety. As a result, we only explore the safe set when necessary for the IML problem. We apply our framework to safe Bayesian optimization and to safe exploration in deterministic Markov Decision Processes (MDP), which have been analyzed separately before. Our method outperforms other algorithms empirically.