Physics-Informed Reward Machines

TL;DR

This paper introduces physics-informed reward machines (pRMs), a symbolic framework that enhances the expressiveness and efficiency of reinforcement learning by enabling complex reward structures and leveraging counterfactual experiences.

Contribution

The paper proposes pRMs, a novel symbolic reward machine framework that incorporates physical knowledge to improve learning speed and expressiveness in RL tasks.

Findings

pRMs accelerate reward learning in physical environments

Counterfactual experience generation improves sample efficiency

pRMs outperform baseline methods in control tasks

Abstract

Reward machines (RMs) provide a structured way to specify non-Markovian rewards in reinforcement learning (RL), thereby improving both expressiveness and programmability. Viewed more broadly, they separate what is known about the environment, captured by the reward mechanism, from what remains unknown and must be discovered through sampling. This separation supports techniques such as counterfactual experience generation and reward shaping, which reduce sample complexity and speed up learning. We introduce physics-informed reward machines (pRMs), a symbolic machine designed to express complex learning objectives and reward structures for RL agents, thereby enabling more programmable, expressive, and efficient learning. We present RL algorithms capable of exploiting pRMs via counterfactual experiences and reward shaping. Our experimental results show that these techniques accelerate reward acquisition during the training phases of RL. We demonstrate the expressiveness and effectiveness of pRMs through experiments in both finite and continuous physical environments, illustrating that incorporating pRMs significantly improves learning efficiency across several control tasks.