Policy Learning for Balancing Short-Term and Long-Term Rewards

TL;DR

This paper introduces a new framework for learning policies that balance short-term and long-term rewards, even with missing long-term data, providing theoretical guarantees and practical algorithms.

Contribution

It formalizes a novel approach to policy learning balancing rewards, establishes identifiability and efficiency bounds, and develops estimators with proven asymptotic properties.

Findings

Proposed estimators are consistent and asymptotically normal.

Short-term outcomes improve long-term reward estimation.

Method demonstrates effective policy learning in experiments.

Abstract

Empirical researchers and decision-makers spanning various domains frequently seek profound insights into the long-term impacts of interventions. While the significance of long-term outcomes is undeniable, an overemphasis on them may inadvertently overshadow short-term gains. Motivated by this, this paper formalizes a new framework for learning the optimal policy that effectively balances both long-term and short-term rewards, where some long-term outcomes are allowed to be missing. In particular, we first present the identifiability of both rewards under mild assumptions. Next, we deduce the semiparametric efficiency bounds, along with the consistency and asymptotic normality of their estimators. We also reveal that short-term outcomes, if associated, contribute to improving the estimator of the long-term reward. Based on the proposed estimators, we develop a principled policy learning approach and further derive the convergence rates of regret and estimation errors associated with the learned policy. Extensive experiments are conducted to validate the effectiveness of the proposed method, demonstrating its practical applicability.