Linear Mixture Distributionally Robust Markov Decision Processes

TL;DR

This paper introduces a linear mixture distributionally robust Markov decision process framework that refines uncertainty modeling and provides theoretical guarantees for robust policy learning under various divergence measures.

Contribution

It proposes a novel linear mixture DRMDP framework with refined uncertainty sets and a meta algorithm for robust policy learning, supported by sample complexity analysis.

Findings

More refined uncertainty representation compared to traditional models.

Sample complexity bounds established for multiple divergence measures.

Theoretical foundation for future research in robust MDPs.

Abstract

Many real-world decision-making problems face the off-dynamics challenge: the agent learns a policy in a source domain and deploys it in a target domain with different state transitions. The distributionally robust Markov decision process (DRMDP) addresses this challenge by finding a robust policy that performs well under the worst-case environment within a pre-specified uncertainty set of transition dynamics. Its effectiveness heavily hinges on the proper design of these uncertainty sets, based on prior knowledge of the dynamics. In this work, we propose a novel linear mixture DRMDP framework, where the nominal dynamics is assumed to be a linear mixture model. In contrast with existing uncertainty sets directly defined as a ball centered around the nominal kernel, linear mixture DRMDPs define the uncertainty sets based on a ball around the mixture weighting parameter. We show that this new framework provides a more refined representation of uncertainties compared to conventional models based on $(s,a)$-rectangularity and $d$-rectangularity, when prior knowledge about the mixture model is present. We propose a meta algorithm for robust policy learning in linear mixture DRMDPs with general $f$-divergence defined uncertainty sets, and analyze its sample complexities under three divergence metrics instantiations: total variation, Kullback-Leibler, and $\chi^2$ divergences. These results establish the statistical learnability of linear mixture DRMDPs, laying the theoretical foundation for future research on this new setting.