Near-Optimal Dynamic Regret for Adversarial Linear Mixture MDPs

TL;DR

This paper introduces a novel algorithm for adversarial linear mixture MDPs that achieves near-optimal dynamic regret without prior knowledge of non-stationarity, effectively handling unknown transitions and non-stationary environments.

Contribution

It proposes a combined occupancy-measure and policy-based algorithm with a new conversion technique, achieving near-optimal regret bounds in adversarial settings.

Findings

Achieves $ ilde{O}(d \, \sqrt{H^3 K} + \sqrt{HK(H + \bar{P}_K)})$ dynamic regret.

Establishes a matching lower bound, proving near-optimality.

First work to attain near-optimal regret without prior non-stationarity knowledge.

Abstract

We study episodic linear mixture MDPs with the unknown transition and adversarial rewards under full-information feedback, employing dynamic regret as the performance measure. We start with in-depth analyses of the strengths and limitations of the two most popular methods: occupancy-measure-based and policy-based methods. We observe that while the occupancy-measure-based method is effective in addressing non-stationary environments, it encounters difficulties with the unknown transition. In contrast, the policy-based method can deal with the unknown transition effectively but faces challenges in handling non-stationary environments. Building on this, we propose a novel algorithm that combines the benefits of both methods. Specifically, it employs (i) an occupancy-measure-based global optimization with a two-layer structure to handle non-stationary environments; and (ii) a policy-based variance-aware value-targeted regression to tackle the unknown transition. We bridge these two parts by a novel conversion. Our algorithm enjoys an $\widetilde{\mathcal{O}}(d \sqrt{H^3 K} + \sqrt{HK(H + \bar{P}_K)})$ dynamic regret, where $d$ is the feature dimension, $H$ is the episode length, $K$ is the number of episodes, $\bar{P}_K$ is the non-stationarity measure. We show it is minimax optimal up to logarithmic factors by establishing a matching lower bound. To the best of our knowledge, this is the first work that achieves near-optimal dynamic regret for adversarial linear mixture MDPs with the unknown transition without prior knowledge of the non-stationarity measure.