Distributionally Robust Policy Learning under Concept Drifts

TL;DR

This paper develops a distributionally robust policy learning framework that specifically addresses concept drift by focusing on changes in the conditional distribution, providing estimators and algorithms with proven optimality and empirical validation.

Contribution

It introduces a doubly-robust estimator for worst-case policy evaluation under concept drift and proposes an optimal policy learning algorithm with theoretical guarantees.

Findings

The estimator is asymptotically normal even with slow nuisance parameter estimation.

The policy learning algorithm achieves a sub-optimality gap of order n^{-1/2}.

Numerical studies show substantial improvements over benchmarks.

Abstract

Distributionally robust policy learning aims to find a policy that performs well under the worst-case distributional shift, and yet most existing methods for robust policy learning consider the worst-case joint distribution of the covariate and the outcome. The joint-modeling strategy can be unnecessarily conservative when we have more information on the source of distributional shifts. This paper studies a more nuanced problem -- robust policy learning under the concept drift, when only the conditional relationship between the outcome and the covariate changes. To this end, we first provide a doubly-robust estimator for evaluating the worst-case average reward of a given policy under a set of perturbed conditional distributions. We show that the policy value estimator enjoys asymptotic normality even if the nuisance parameters are estimated with a slower-than-root-$n$ rate. We then propose a learning algorithm that outputs the policy maximizing the estimated policy value within a given policy class $\Pi$, and show that the sub-optimality gap of the proposed algorithm is of the order $\kappa(\Pi)n^{-1/2}$, where $\kappa(\Pi)$ is the entropy integral of $\Pi$ under the Hamming distance and $n$ is the sample size. A matching lower bound is provided to show the optimality of the rate. The proposed methods are implemented and evaluated in numerical studies, demonstrating substantial improvement compared with existing benchmarks.