From Relative Entropy to Minimax: A Unified Framework for Coverage in MDPs

TL;DR

This paper introduces a unified framework for exploration in reward-free MDPs using a family of concave coverage objectives, enabling explicit control over exploration priorities and recovering worst-case strategies as a parameter grows.

Contribution

It proposes a novel family of coverage objectives over occupancy measures that unifies existing approaches and provides a gradient-based method for targeted exploration in MDPs.

Findings

The framework unifies divergence-based, weighted, and minimax coverage objectives.

The gradient-based algorithm effectively directs exploration towards under-explored state-action pairs.

Increasing the parameter emphasizes worst-case coverage, recovering minimax strategies.

Abstract

Targeted and deliberate exploration of state--action pairs is essential in reward-free Markov Decision Problems (MDPs). More precisely, different state-action pairs exhibit different degree of importance or difficulty which must be actively and explicitly built into a controlled exploration strategy. To this end, we propose a weighted and parameterized family of concave coverage objectives, denoted by $U_\rho$, defined directly over state--action occupancy measures. This family unifies several widely studied objectives within a single framework, including divergence-based marginal matching, weighted average coverage, and worst-case (minimax) coverage. While the concavity of $U_\rho$ captures the diminishing return associated with over-exploration, the simple closed form of the gradient of $U_\rho$ enables an explicit control to prioritize under-explored state--action pairs. Leveraging this structure, we develop a gradient-based algorithm that actively steers the induced occupancy toward a desired coverage pattern. Moreover, we show that as $\rho$ increases, the resulting exploration strategy increasingly emphasizes the least-explored state--action pairs, recovering worst-case coverage behavior in the limit.