Is Pure Exploitation Sufficient in Exogenous MDPs with Linear Function Approximation?

TL;DR

This paper demonstrates that in exogenous MDPs, pure exploitation algorithms can achieve strong regret bounds without explicit exploration, challenging the traditional belief that exploration is necessary.

Contribution

The paper introduces PEL and LSVI-PE algorithms with the first finite-sample regret guarantees for exploitation-only methods in Exo-MDPs, using novel analytical tools.

Findings

PEL achieves $ ilde{O}(H^2|\Xi|\sqrt{K})$ regret in tabular case

LSVI-PE handles large continuous spaces with polynomial regret

Exploitation-only methods outperform baselines in experiments

Abstract

Exogenous MDPs (Exo-MDPs) capture sequential decision-making where uncertainty comes solely from exogenous inputs that evolve independently of the learner's actions. This structure is especially common in operations research applications such as inventory control, energy storage, and resource allocation, where exogenous randomness (e.g., demand, arrivals, or prices) drives system behavior. Despite decades of empirical evidence that greedy, exploitation-only methods work remarkably well in these settings, theory has lagged behind: all existing regret guarantees for Exo-MDPs rely on explicit exploration or tabular assumptions. We show that exploration is unnecessary. We propose Pure Exploitation Learning (PEL) and prove the first general finite-sample regret bounds for exploitation-only algorithms in Exo-MDPs. In the tabular case, PEL achieves $\widetilde{O}(H^2|\Xi|\sqrt{K})$. For large, continuous endogenous state spaces, we introduce LSVI-PE, a simple linear-approximation method whose regret is polynomial in the feature dimension, exogenous state space, and horizon, independent of the endogenous state and action spaces. Our analysis introduces two new tools: counterfactual trajectories and Bellman-closed feature transport, which together allow greedy policies to have accurate value estimates without optimism. Experiments on synthetic and resource-management tasks show that PEL consistently outperforming baselines. Overall, our results overturn the conventional wisdom that exploration is required, demonstrating that in Exo-MDPs, pure exploitation is enough.