Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learning

TL;DR

This paper introduces cascaded gaps for risk-sensitive reinforcement learning, enabling gap-dependent regret bounds that significantly improve over previous gap-independent bounds, with theoretical guarantees and near-optimal lower bounds.

Contribution

It proposes a new concept of cascaded gaps for risk-sensitive RL and derives tight regret bounds that adapt to problem structure, improving over existing methods.

Findings

Non-asymptotic logarithmic regret bounds derived

Exponential improvement over gap-independent bounds shown

Lower bounds certify near-optimality of the proposed bounds

Abstract

In this paper, we study gap-dependent regret guarantees for risk-sensitive reinforcement learning based on the entropic risk measure. We propose a novel definition of sub-optimality gaps, which we call cascaded gaps, and we discuss their key components that adapt to the underlying structures of the problem. Based on the cascaded gaps, we derive non-asymptotic and logarithmic regret bounds for two model-free algorithms under episodic Markov decision processes. We show that, in appropriate settings, these bounds feature exponential improvement over existing ones that are independent of gaps. We also prove gap-dependent lower bounds, which certify the near optimality of the upper bounds.