DiPRL: Learning Discrete Programmatic Policies via Architecture Entropy Regularization

TL;DR

DiPRL is a novel reinforcement learning method that learns nearly discrete, interpretable programmatic policies during training, avoiding performance loss from post-hoc discretization.

Contribution

It introduces architecture entropy regularization to enable smooth training of nearly discrete policies without needing post-hoc fine-tuning.

Findings

DiPRL achieves strong performance on multiple RL tasks.

It maintains policy interpretability with nearly discrete programs.

The method mitigates performance drops associated with post-hoc discretization.

Abstract

Programmatic reinforcement learning (PRL) offers an interpretable alternative to deep reinforcement learning by representing policies as human-readable and -editable programs. While gradient-based methods have been developed to optimize continuous relaxations of programs, they face a significant performance drop when converting the continuous relaxations back into discrete programs. Post-hoc discretization can discard optimized branches and parameters in a program, which results in a collapse of policy expressivity and lowered task performance, leading in turn to a need for additional fine-tuning. To overcome these limitations, we propose Differentiable Discrete Programmatic Reinforcement Learning (DiPRL), a method that learns programmatic policies that become nearly discrete during training, avoiding a separate post-hoc fine-tuning stage. We first analyze the inherent risks of performance drop introduced by post-hoc discretization of gradient-based methods. Then, we introduce programmatic architecture entropy regularization, which enables smooth, differentiable training that encourages convergence toward a discrete program. DiPRL maintains the efficiency of gradient-based optimization while mitigating the risks of post-hoc discretization. Our experiments across multiple discrete and continuous RL tasks demonstrate that DiPRL can achieve strong performance via interpretable programmatic policies.