Programmatic Policy Extraction by Iterative Local Search

TL;DR

This paper introduces a straightforward method for extracting interpretable, programmatic policies from neural network policies using iterative local search, demonstrated on control tasks like pendulum swing-up.

Contribution

The paper presents a novel local search heuristic for directly extracting simple, interpretable policies from pretrained neural networks, improving interpretability and verification.

Findings

Extracted policies are simple and interpretable.

Performance is close to the original neural policies.

Applicable to policies trained by imitation or neural methods.

Abstract

Reinforcement learning policies are often represented by neural networks, but programmatic policies are preferred in some cases because they are more interpretable, amenable to formal verification, or generalize better. While efficient algorithms for learning neural policies exist, learning programmatic policies is challenging. Combining imitation-projection and dataset aggregation with a local search heuristic, we present a simple and direct approach to extracting a programmatic policy from a pretrained neural policy. After examining our local search heuristic on a programming by example problem, we demonstrate our programmatic policy extraction method on a pendulum swing-up problem. Both when trained using a hand crafted expert policy and a learned neural policy, our method discovers simple and interpretable policies that perform almost as well as the original.