Scalable Semantic Non-Markovian Simulation Proxy for Reinforcement Learning

TL;DR

This paper introduces a scalable, explainable semantic simulation proxy for reinforcement learning that handles non-Markovian dynamics, significantly improving speed while maintaining policy quality.

Contribution

It proposes a novel semantic proxy based on annotated logic, enabling fast, non-Markovian simulation with explainability, addressing key RL scalability and interpretability issues.

Findings

Achieves up to 1000x speed-up compared to high-fidelity simulators.

Preserves policy quality despite increased simulation speed.

Models non-Markovian dynamics and instantaneous actions effectively.

Abstract

Recent advances in reinforcement learning (RL) have shown much promise across a variety of applications. However, issues such as scalability, explainability, and Markovian assumptions limit its applicability in certain domains. We observe that many of these shortcomings emanate from the simulator as opposed to the RL training algorithms themselves. As such, we propose a semantic proxy for simulation based on a temporal extension to annotated logic. In comparison with two high-fidelity simulators, we show up to three orders of magnitude speed-up while preserving the quality of policy learned. In addition, we show the ability to model and leverage non-Markovian dynamics and instantaneous actions while providing an explainable trace describing the outcomes of the agent actions.