Offline Policy Learning with Weight Clipping and Heaviside Composite Optimization

TL;DR

This paper introduces a novel offline policy learning method that uses weight clipping to reduce variance in policy value estimates, reformulates the optimization as a Heaviside composite problem, and employs integer programming for efficient solution.

Contribution

It develops a weight-clipping estimator for offline policy learning, reformulates the optimization as a Heaviside composite problem, and provides theoretical bounds on suboptimality.

Findings

Weight clipping reduces variance in policy value estimation.

Reformulation as Heaviside composite optimization enables efficient solving.

The proposed method improves policy learning performance theoretically.

Abstract

Offline policy learning aims to use historical data to learn an optimal personalized decision rule. In the standard estimate-then-optimize framework, reweighting-based methods (e.g., inverse propensity weighting or doubly robust estimators) are widely used to produce unbiased estimates of policy values. However, when the propensity scores of some treatments are small, these reweighting-based methods suffer from high variance in policy value estimation, which may mislead the downstream policy optimization and yield a learned policy with inferior value. In this paper, we systematically develop an offline policy learning algorithm based on a weight-clipping estimator that truncates small propensity scores via a clipping threshold chosen to minimize the mean squared error (MSE) in policy value estimation. Focusing on linear policies, we address the bilevel and discontinuous objective induced by weight-clipping-based policy optimization by reformulating the problem as a Heaviside composite optimization problem, which provides a rigorous computational framework. The reformulated policy optimization problem is then solved efficiently using the progressive integer programming method, making practical policy learning tractable. We establish an upper bound for the suboptimality of the proposed algorithm, which reveals how the reduction in MSE of policy value estimation, enabled by our proposed weight-clipping estimator, leads to improved policy learning performance.