Advantage Amplification in Slowly Evolving Latent-State Environments

TL;DR

This paper introduces advantage amplification techniques using temporal abstraction to improve reinforcement learning in long-horizon, latent-state environments like recommender systems, addressing key challenges such as belief state error.

Contribution

It proposes a general principle of advantage amplification, develops aggregation methods, and provides theoretical and empirical validation for slowly evolving latent-state environments.

Findings

Advantage amplification can mitigate belief state errors.

Aggregation methods induce advantage amplification under certain conditions.

Empirical results show improved performance in user-modeling tasks.

Abstract

Latent-state environments with long horizons, such as those faced by recommender systems, pose significant challenges for reinforcement learning (RL). In this work, we identify and analyze several key hurdles for RL in such environments, including belief state error and small action advantage. We develop a general principle of advantage amplification that can overcome these hurdles through the use of temporal abstraction. We propose several aggregation methods and prove they induce amplification in certain settings. We also bound the loss in optimality incurred by our methods in environments where latent state evolves slowly and demonstrate their performance empirically in a stylized user-modeling task.

Figures

Equations

$$V^{\pi }(\mathfrak{b})=\underset{a\sim \pi (\mathfrak{b})}{E}\left[R(\mathfrak{b},a)+\gamma \sum _{z\in \mathcal{Z}}\mathrm{Pr}(z|\mathfrak{b},a)V^{\pi }(U(\mathfrak{b},a,z))\right]$$

$${\left|p(s_{t+1}|H_t)-p(s_{t+1}|\varphi (H_t))\right|}_{\mathrm{TV}}<\epsilon ,$$

$$\left|V^{\ast }(\varphi (H))-\hat{V}(\varphi (H))\right|\le \frac{2\epsilon r_{\text{max}}}{(1-\gamma {)}^3}.$$

$$\mathfrak{S}\triangleq \frac{{\sup }_{\mathfrak{b}}A(\mathfrak{b})}{{\sup }_{\mathfrak{b}:A(\mathfrak{b})\le 2\epsilon r_{\max }/(1-\gamma {)}^2}A(\mathfrak{b})}-1,$$

$$|Q^{\ast }(\mathfrak{b},a)-Q^{\ast }({\mathfrak{b}}^{\prime },a)|\le L.$$

$$\delta =\frac{\epsilon Q_{\max }}{1-\gamma }\le \frac{\epsilon r_{\max }}{(1-\gamma {)}^2}.$$

$$Q^{\times k\ast }$$

$$\ge$$

$${\epsilon }_{\max }\le L\frac{k(\gamma -{\gamma }^k)-\gamma (1-(1+k-\gamma k){\gamma }^k)}{r_{\max }}$$

$${\pi }^{\ast }=\underset{\pi }{\mathrm{argmax}}\sum _t{\gamma }^t{\mathbb{E}}_{\pi }\left(R_t-T\cdot \text{mathbbm}1[a_t\ne a_{t-1}]\right).$$

$$\kappa =\frac{\log \gamma +(\gamma -1)W\left(\frac{{\gamma }^{1/(1-\gamma )}}{\gamma -1}\left(\frac{(1-\gamma {)}^2}{2\gamma L}T-1\right)\log \gamma \right)}{(\gamma -1)\log \gamma },$$

$$V^{\pi }(\mathfrak{b})-$$

$$=Q^{\pi }(\mathfrak{b},\pi (\mathfrak{b}))-Q^{\rho }(\mathfrak{b},\rho (\mathfrak{b}))$$

$$=\underset{\rho }{E}\left[\sum _{i=0}{\gamma }^i\left(Q^{\pi }({\mathfrak{b}}_i,\pi ({\mathfrak{b}}_i))-Q^{\pi }({\mathfrak{b}}_i,\rho ({\mathfrak{b}}_i))\right)\right],$$

$$Q^{\pi }(\mathfrak{b},\pi (\mathfrak{b}))$$

$$=$$

$$+Q^{\rho \pi }(\mathfrak{b},\rho \pi (\mathfrak{b}))-Q^{\rho }(\mathfrak{b},\rho (\mathfrak{b}))$$

$$=$$

$$+Q^{\pi }(\mathfrak{b},\rho (\mathfrak{b}))-Q^{\rho }(\mathfrak{b},\rho (\mathfrak{b}))$$

$$=$$

$$+\gamma \underset{{\mathfrak{b}}^{\prime }\sim \rho }{E}[Q^{\pi }({\mathfrak{b}}^{\prime },\pi ({\mathfrak{b}}^{\prime }))-Q^{\rho }({\mathfrak{b}}^{\prime },\rho ({\mathfrak{b}}^{\prime }))].$$

$$Q^{\ast }(\mathfrak{b},a^{\times k})-$$

$$=E\left[\sum _{i=0}^k{\gamma }^i\left(Q^{\ast }({\mathfrak{b}}_i^{a^{\prime }},a)-Q^{\ast }({\mathfrak{b}}_i^{a^{\prime }},a^{\prime })\right)\right]$$

$$A^{\times k}(\mathfrak{b}):=$$

$$\ge$$

$$-2\gamma L\frac{1-(1+k-\gamma k){\gamma }^k}{(1-\gamma {)}^2},$$

$$A^{\times k}(\mathfrak{b})$$

$$\ge E\left[\sum _{i=0}^k{\gamma }^i(A(\mathfrak{b})-2iL)\right]$$

$$=A(\mathfrak{b})\frac{1-{\gamma }^k}{1-\gamma }-2L\frac{\gamma -(1+k-\gamma k){\gamma }^{k+1}}{(1-\gamma {)}^2}.$$

$$\sum _{i=0}^{k}i{\gamma }^i=\frac{\gamma -(1+k-\gamma k){\gamma }^{k+1}}{(1-\gamma {)}^2}.$$

$$V^{\pi }(\mathfrak{b})-$$

$$=\underset{\rho }{E}\left[\sum _{i=0}{\gamma }^i\left(Q^{\pi }({\mathfrak{b}}_i,\pi ({\mathfrak{b}}_i))-Q^{\pi }({\mathfrak{b}}_i,\rho ({\mathfrak{b}}_i))\right)\right].$$

$$V^{\pi }(\mathfrak{b})-V^{\rho }(\mathfrak{b})\le \sum _{i=0}{\gamma }^{i}2kL=\frac{2kL}{1-\gamma }.$$

$$A(\mathfrak{b}{)}^{\times k}$$

$$\ge$$

$$-\frac{2kL}{1-\gamma }.$$

$$Q^{\times k\ast }(\mathfrak{b},a^{\times k})\ge Q^{\ast }(\mathfrak{b},a)-\frac{2kL}{1-\gamma }={\bar{Q}}^{\times k\ast }(\mathfrak{b},a^{\times k})-\frac{2kL}{1-\gamma }.$$

$$Q^{\times k\ast }(\mathfrak{b},a^{\prime \times k})\le {\bar{Q}}^{\times k\ast }(\mathfrak{b},a^{\prime \times k}),$$

$$Q^{\times k\ast }(\mathfrak{b},a^{\times k})$$

Full text

1 Introduction

Long-term value (LTV) estimation and optimization is of increasing importance in the design of recommender systems (RSs), and other user-facing systems. Often the problem is framed as a Markov decision process (MDP) and solved using MDP algorithms or reinforcement learning (RL) Shani et al. (2005); Taghipour et al. (2007); Choi et al. (2018); Zhao et al. (2017); Archak et al. (2012); Mladenov et al. (2017). Typically, actions are the set of recommendable items111Item slates are often recommended, but we ignore this here.; states reflect information about the user (e.g., static attributes, past interactions, context/query); and rewards measure some form of user engagement (e.g., clicks, views, time spent, purchase). Such event-level models have seen some success, but current state-of-the-art is limited to very short horizons.

When dealing with long-term user behavior, it is vital to consider the impact of recommendations on user latent state (e.g., satisfaction, latent interests, or item awareness) which often governs both immediate and long-term behavior. Indeed, the main promise of using RL/MDP models for RSs is to (a) identify latent state (e.g., uncover topic interests via exploration) and (b) influence the latent state (e.g., create new interests or improve awareness and satisfaction). That said, evidence is emerging that at least some aspects of user latent state evolve very slowly. For example, Hohnhold et al. Hohnhold et al. (2015) show that varying ad quality and ad load induces slow, but inexorable (positive or negative) changes in user click propensity over a period of months, while Wilhelm et al. Wilhelm et al. (2018) show that explicitly diversifying recommendations in YouTube induces similarly slow, persistent changes in user engagement (see such slow “user learning” curves in Fig. 1).

Event-level RL in such settings is challenging for several reasons. First, the effective horizon over which an RS policy influences the latent state can extend up to $O(10^{4}\textup{--}10^{5})$ state transitions. Indeed, the cumulative effect of recommendations is vital for LTV optimization, but the long-term impact of any single recommendation is often dwarfed by immediate reward differences. Second, the MDP is partially observable, requiring some form of belief state estimation. Third, the impact of latent state on immediate observable behavior is often small and very noisy—the problems have a low signal-to-noise ratio (SNR). We detail below how these factors interact.

Given the importance of LTV optimization in RSs, we propose a new technique called advantage amplification to overcome these challenges. Intuitively, amplification seeks to overcome the error induced by state estimation by introducing (explicit or implicit) temporal abstraction across policy space. We require that policies take a series of actions, thus allowing more accurate value estimation by mitigating the cumulative effects of state-estimation error. We first consider temporal aggregation, where an action is held fixed for a short horizon. We show that this can lead to significant amplification of the advantage differences between abstract actions (relative to event-level actions). This is a form of MDP/RL temporal abstraction as used in hierarchical RL Sutton et al. (1999); Barto and Mahadevan (2003) and can be viewed as options or macros designed for the purpose of allowing distinction of good and bad behaviors in latent-state domains with low SNR (rather than, say, for subgoal achievement). We generalize this by analyzing policies with (artificial) action switching costs, which induces similar amplification with more flexibility.

Limiting policies to temporally abstract actions induces potential sub-optimality Parr (1998); Hauskrecht et al. (1998). However, since the underlying latent state often evolves slowly w.r.t. the event horizon in RS settings, we identify a “smoothness” property that is used to bound the induced error of advantage amplification.

Our contributions are as follows. We introduce a stylized model of slow user learning in RSs (Sec. 2) and formalize this as a POMDP (Sec. 3), defining several novel concepts, and show how low SNR interacts poorly with belief-state approximation (Sec. 4.1). We develop advantage amplification as a principle and prove that action aggregation (Sec. 4.2) and switching cost regularization (Sec. 4.3) provide strong amplification guarantees with minimal policy loss under suitable conditions. Experiments with stylized models show the effectiveness of these techniques.222Proofs, auxiliary lemmas and additional experiments are available in an extended version of the paper.

2 User Satisfaction: An Illustrative Example

Before formalizing our problem, we describe a stylized model reflecting the dynamics of user satisfaction as a user interacts with an RS. The model is intentionally stylized to help illustrate key concepts underlying the formal model and analysis developed in the sequel (hence ignores much of the true complexity of user satisfaction). Though focused on user-RS engagement, the principles apply more broadly to any latent-state system with low SNR and slowly evolving latent state.

Our model captures the relationship between a user and an RS over an extended period (e.g., a content recommender of news, video, or music) through overall user satisfaction, which is not known to the RS. We hypothesize that satisfaction is one (of several) key latent factors that impacts user engagement; and since new treatments often induce slow-moving or delayed effects on user behavior, we assume this latent variable evolves slowly as a function of the quality of the content consumed Hohnhold et al. (2015) (and see Fig. 1 (left)). Finally, the model captures the tension between (often low-quality) content that encourages short-term engagement (e.g., manipulative, provocative or distracting content) at the expense of long-term engagement; and high-quality content that promotes long-term usage but can sacrifice near-term engagement.

Our model includes two classes of recommendable items. Some items induce high immediate engagement, but degrade user engagement over the long run. We dub these “Chocolate” (Choc)—immediately appealing but not very “nutritious.” Other items—dubbed “Kale,” less attractive, but more “nutritious”—induce lower immediate engagement but tend to improve long-term engagement.333Our model allows a real-valued continuum of items (e.g., degree of Choc between $[0,1]$ as in our experiments) like measures of ad quality. We use the binary form to streamline our initial exposition. We call this the Choc-Kale model (CK). A stationary, stochastic policy can be represented by a single scalar $0\leq\pi\leq 1$ representing the probability of taking action Choc. We sometimes refer to Choc as a “negative” and Kale as a “positive” recommendation.

We use a single latent variable $s\in[0,1]$ to capture a user’s overall satisfaction with the RS. Satisfaction is driven by net positive exposure $p$, which measures total positive-less-negative recommendations, with a discount $0\leq\beta<1$ applied to ensure that $p$ is bounded: $p\in\left[\frac{-1}{1-\beta},\frac{1}{1-\beta}\right]$. We view $p$ as a user’s learned perception of the RS and $s$ as how this influences gradual changes in engagement.

A user response to a recommendation $a$ is her degree of engagement $g(s,a)$, and depends stochastically on both the quality of the recommendation, and her latent state $s$. $g$ is a random function, e.g., responses might be normally distributed: $g(s,a)\sim N(s\cdot\mu_{a},\sigma_{a}^{2})~{}~{}\text{for }a\in\{\textit{Choc},\textit{Kale}\}$. We use $g(s,a)$ also to denote expected engagement. We require that Choc results in greater immediate (expected) engagement than Kale, $g(s,\textit{Kale})<g(s,\textit{Choc})$, for any fixed $s$.

The dynamics of $p$ is straightforward. A Kale exposure increases $p$ by 1 and Choc decreases it by 1 (with discounting): $p_{t+1}\leftarrow\beta p_{t}+1$ with Kale (and $-1$ with Choc). Satisfaction $s$ is a user-learned function of $p$ and follows a sigmoidal learning curve: $s(p)=1/(1+e^{-\tau p})$, where $\tau$ is a temperature/learning rate parameter. Other learning curves are possible, but the sigmoidal model captures both positive and negative exponential learning as hypothesized in psychological-learning literature Thurstone (1919); Jaber (2006) and as observed in the empirical curves in Fig. 1.444Such learning curves are often reflective of aggregate behavior, obscuring individual differences that are much less “smooth.” However, unless cues are available that allow us to model such individual differences, the aggregate model serves a valuable role even when optimizing for individual users.

We compute the Q-values of Choc and Kale for each satisfaction level $s$ and plot them in Fig. 2(a). We observe that when satisfaction is low, Kale is a better recommendation, and above some level Choc becomes preferable, as expected. We also see that for any $s$ the difference in Q-values is rather small. With additional noise, the Q-values become practically indistinguishable for a large range of satisfaction levels (Fig. 2(b)), which illustrates the hardness of RL in this setting.

3 Problem Statement

We outline a basic latent-state control problem as a partially observable MDP (POMDP) that encompasses the notions above. We highlight several properties that play a key role in the analysis of latent-state RL we develop in the next section.

We consider environments that can be modeled as a POMDP $\mathcal{M}=\langle\mathcal{S},\mathcal{A},T,\mathcal{Z},O,R,\mathfrak{b}_{0},\gamma\rangle$ Smallwood and Sondik (1973). States $\mathcal{S}$ reflect user latent state and other observable aspects of the domain: in the CK model, this is simply a user’s current satisfaction $s$. Actions $\mathcal{A}$ are recommendable items: in CK, we distinguish only Choc from Kale. The transition kernel $T(s,a,s^{\prime})$ in the CK model is $T(s^{\prime},a,s)=1$ if $s^{\prime}=(1+\exp(\beta\log\left(1-1/s\right)-\beta\tau a))^{-1}$, where $a$ is 1 (resp., -1) for action Kale (resp., Choc).555This is easily randomized if desired. Observations $\mathcal{Z}$ reflect observable user behavior and $O(s,a,z)$ the probability of $z\in\mathcal{Z}$ when $a$ is taken at state $s$. In CK, $\mathcal{Z}$ is the observed engagement with a recommendation while $O$ reflects the random realization of $g(s,a)$. The immediate reward $R(s,a)$ is (expected) user engagement (we let $r_{\max}=\max_{s,a}R(s,a)$), $\mathfrak{b}_{0}$ the initial state distribution, and $\gamma\in[0,1)$ the discount factor.

In this POMDP, an RS does not have access to the true state $s$, but must generate policies that depend only on the sequence of past action-observation pairs—let ${\cal H}^{\ast}$ be the set of all finite such sequences $(a_{t},z_{t})_{t\in\mathbb{N}}$. Any such history can be summarized, via optimal Bayes filtering, as a distribution or belief state $\mathfrak{b}\in\Delta(\mathcal{S})$. More generally, this “belief state” can be any summarization of ${\mathcal{H}}^{\ast}$ used to make decisions. It may be, say, a collection of sufficient statistics, or a deep recurrent embedding of history. Let $\mathcal{B}$ denote the set of (realizable) belief states. We also require a mapping $U:\mathcal{B}\times\mathcal{A}\times\mathcal{Z}\rightarrow\mathcal{B}$ that describes the update $U(\mathfrak{b},a,z)$ of any $\mathfrak{b}\in\mathcal{B}$ given $a\in\mathcal{A},z\in\mathcal{Z}$. The pair $(\mathcal{B},U)$ defines our representation.

A (stochastic) policy is a mapping $\pi:\mathcal{B}\rightarrow\Delta(\mathcal{A})$ that selects an action distribution $\pi(\mathfrak{b})$ for execution given belief $\mathfrak{b}$; we write $\pi(a|\mathfrak{b})$ to indicate the probability of action $a$. Deterministic policies are defined in the usual way. The value of a policy $\pi$ is given by the standard recurrence:666Here $R(\mathfrak{b},a)$ and $\Pr(z|\mathfrak{b},a)$ are given by expectations of $R$ and $O$, respectively, w.r.t. $s\sim\mathfrak{b}$ if $\mathfrak{b}\in\Delta(\mathcal{S})$. The interpretation for other representations is discussed below.

[TABLE]

We define $Q^{\pi}(\mathfrak{b},a)$ by fixing $a$ in Eq. 1 (rather than taking an expectation). An optimal policy $\pi^{\ast}=\sup V^{\pi}$ over $\mathcal{B}$ has value (resp., Q) function $V^{\ast}$ (resp., $Q^{\ast}$). Optimal policies and values can be computed using dynamic programming or learned using (partially observable) RL methods. When we learn a Q-function $Q$, whether exactly or approximately, the policy induced by $Q$ is the greedy policy $\pi(\mathfrak{b})=\arg\max_{a}Q(\mathfrak{b},a)$ and its induced value function is $V(\mathfrak{b})=\max_{a}Q(\mathfrak{b},a)=Q(\mathfrak{b},a^{\ast}(\mathfrak{b}))$. The advantage function $A(a,\mathfrak{b})=V^{\ast}(\mathfrak{b})-Q^{\ast}(\mathfrak{b},a)$ reflects the difference in the expected value of doing $a$ at $\mathfrak{b}$ (and then acting optimally) vs. acting optimally at $\mathfrak{b}$ Baird III (1999). If $a_{2}$ is the second-best action at $\mathfrak{b}$, the advantage of that belief state is $A(\mathfrak{b})=V^{\ast}(\mathfrak{b})-Q^{\ast}(\mathfrak{b},a_{2})$.

Eq. 1 assumes optimal Bayesian filtering, i.e., the representation $(\mathcal{B},U)$ must be such that the (implicit) expectations over $R$ and $O$ are exact for any history that maps to $\mathfrak{b}$. Unfortunately, exact recursive state estimation is intractable, except for special cases (e.g., linear-Gaussian control). As a consequence, approximation schemes are used in practice (e.g., variational projections Boyen and Koller (1998); fixed-length histories, incl. treating observations as state Singh et al. (1994); learned PSRs Littman and Sutton (2002); recursive policy/Q-function representations Downey et al. (2017)). Approximate histories render the process non-Markovian; as such, a counterfactually estimated Q-value of a policy (e.g., using offline data) differs from its true value due to modified latent-state dynamics (not reflected in the data). In this case, any RL method that treats $\mathfrak{b}$ as (Markovian) state induces a suboptimal policy. We can bound the induced suboptimality using $\varepsilon$-sufficient statistics Francois-Lavet et al. (2017). A function $\phi:{\cal H}^{\ast}\rightarrow\mathcal{B}$ is an $\varepsilon$-sufficient statistic if, for all $H_{t}\in{\cal H}^{\ast}$,

[TABLE]

where $|\cdot|_{\mathrm{TV}}$ is the total variation distance. If $\phi$ is $\varepsilon$-sufficient, then any MDP/RL algorithm that constructs an “optimal” value function $\hat{V}$ over $\mathcal{B}$ incurs a bounded loss w.r.t. $V^{\ast}$ Francois-Lavet et al. (2017):

[TABLE]

The errors in Q-value estimation induced by limitations of $\mathcal{B}$ are irresolvable (i.e., they are a form of model bias), in contrast to error induced by limited data. Moreover, any RL method relying only on offline data is subject to the above bound, regardless of whether the Q-values are estimated directly or not. The impact of this error on model performance can be related to certain properties of the underlying domain as we outline below. A useful quantity for this purpose is the signal-to-noise ratio (SNR) of a POMDP, defined as:

[TABLE]

(the denominator is treated as [math] if no $\mathfrak{b}$ meets the condition).

As discussed above, many aspects of user latent state, such as satisfaction, evolve slowly. We say a POMDP is $L$-smooth if, for all $\mathfrak{b},\mathfrak{b}^{\prime}\in\mathcal{B}$, and $a\in A$ s.t. $T(\mathfrak{b}^{\prime},a,\mathfrak{b})>0$, we have

[TABLE]

Smoothness ensures that for any state reachable under an action $a$, the optimal Q-value of $a$ does not change much.

4 Advantage Amplification

We now detail how low SNR causes difficulty for RL in POMDPs, especially with long horizons (Sec. 4.1). We introduce the principle of advantage amplification to address it (Sec. 4.2) and describe two realizations, temporal aggregation (Sec. 4.2) and switching cost (Sec. 4.3).

4.1 The Impact of Low SNR on RL

The bound Eq. (2) can help assess the impact of low SNR on RL. Assume that policies, values or Q-functions are learned using an approximate belief representation $(\mathcal{B},U)$ that is $\varepsilon$-sufficient. We first show that the error induced by $(\mathcal{B},U)$ is tightly coupled to optimal action advantages in the domain.

Consider an RL agent that learns Q-values using a behavior (data-generating) policy $\rho$. The non-Markovian nature of $(\mathcal{B},U)$ means that: (a) the resulting estimated-optimal policy $\pi$ will have estimated values $\hat{Q}^{\pi}$ that differ from its true values $Q^{\pi}$; and (b) the estimates $\hat{Q}^{\pi}$ (hence, the choice of $\pi$ itself) will depend on $\rho$. We bound the loss of $\pi$ w.r.t. the optimal $\pi^{\ast}$ (with exact filtering) as follows. First, for any (belief) state-action pair $(\mathfrak{b},a)$, suppose the maximum difference between its inferred and optimal Q-values is bounded for any $\rho$: $|Q^{\ast}(\mathfrak{b},a)-Q^{\pi}(\mathfrak{b},a)|\leq\delta$. By Eq. (2) we set

[TABLE]

If $\mathfrak{b}$ has small advantage $A(\mathfrak{b})\leq 2\delta$, under behavior policy $\rho$, the estimate $\hat{Q}(\mathfrak{b},a_{2})$ (the second-best action) can exceed that of $\hat{Q}(\mathfrak{b},a^{\ast}(\mathfrak{b}))$; hence $\pi$ executes $a_{2}$. If $\pi$ visits $\mathfrak{b}$ (or states with similarly small advantages) at a constant rate, the loss w.r.t. $\pi^{\ast}$ compounds, inducing $O(\frac{2\delta}{1-\gamma})$ error.

The tightness of the second part of the argument depends on the structure of the advantage function $A(\mathfrak{b})$. To illustrate, consider two extreme regimes. First, if $A(\mathfrak{b})\geq 2\delta$ at all $\mathfrak{b}\in\mathcal{B}$, i.e., if SNR $\mathfrak{S}=\infty$, state estimation error has no impact on the recovered policy and incurs no loss. In the second regime, if all $A(\mathfrak{b})$ are less than (but on the order of) $2\delta$, i.e., if $\mathfrak{S}=0$, then the inequality is tight provided $\rho$ saturates the state-action error bound. We will see below that low-SNR environments with long horizons (e.g., practical RSs, the stylized CK model) often have such small (but non-trivial) advantages across a wide range of state space.

The latter situation in illustrated on Fig. 2. In Fig. 2(a), the Q-values of the CK model are plotted against the level of satisfaction (as if satisfaction were fully observable). The small advantages are notable. Fig. 2(b) shows the Q-value estimates for $10$ independent tabular Q-learning reruns (the thin lines show the individual runs, the thick lines show the average) where noise is added to $s$. The corrupted $Q$-values at all but the highest satisfaction levels are essentially indistinguishable, leading to extremely poor policies.

4.2 Temporal Abstraction: Action Aggregation

There is a third regime in which state error is relatively benign. Suppose the advantage at each state $\mathfrak{b}$ is either small, $A(\mathfrak{b})\leq\sigma$, or large, $A(\mathfrak{b})>\Sigma$ for some constants $\sigma\ll 2\delta\leq\Sigma$. The induced policy incurs a loss of $\sigma$ at small-advantage states, and no loss on states with large advantages. This leads to a compounded loss of at most $\frac{\sigma}{1-\gamma}$, which may be much smaller than the $\frac{\varepsilon r_{\mathrm{max}}}{(1-\gamma)^{2}}$ error (Eq. 3) depending on $\sigma$.

If the environment is smooth, action aggregation can be used to restructure a problem falling in the second regime to one in this third regime, with $\sigma$ depending on the level of smoothness. This can significantly reduce the impact of estimation error on policy quality by turning the problem into one that is essentially Markovian. More specifically, if at state $\mathfrak{b}$, we know that the optimal (stationary) policy takes action $a$ for the next $k$ decision periods, we consider a reparameterization ${\cal M}^{\times k}$ of the belief-state MDP where, at $\mathfrak{b}$, all actions are executed $k$ times in a row, no matter what the subsequent $k$ states are. In this new problem, the Q-value of the optimal repeated action $Q^{\ast}(\mathfrak{b},a^{\times k})$ is the same as that of its event-level counterpart $Q^{\ast}(\mathfrak{b},a)$, since the same sequence of expected rewards will be generated. Conversely, all suboptimal actions incur a cumulative reduction in Q-value in ${\cal M}^{\times k}$ since their suboptimality compounds over $k$ periods. Thus, in ${\cal M}^{\times k}$, the optimal policy $\pi^{\times k\ast}$ generates the same cumulative discounted return as the event-level optimal policy, while the advantage of $a^{\times k}$ over any other repeated action $a^{\prime\times k}$ at $\mathfrak{b}$ is larger than that of $a$ over $a^{\prime}$ in the event-level problem.

To derive bounds, note that, for an $L$-smooth POMDP, at any state where the advantage is at least $2kL$, the optimal action persists for the next $k$ periods (its Q-value can decrease by at most $L$ while that of the second-best can at most increase by $L$). If we apply aggregation only at such states, the advantage increases to some value $\Sigma$, putting us in regime 3 (i.e., the advantage is either less than $\sigma=2kL$ or more than $\Sigma$). Of course, we cannot “cherry-pick” only states with high advantage for aggregation; but aggregating over all states induces some loss due to the inability to switch actions quickly. We bound that cost in computing $\sigma$ and $\Sigma$. This allows us to first lower bound the regret of the best $k$-aggregate policy:

Theorem 1**.**

*Let $k$ be a fixed horizon, and let $Q^{\ast}$—the event-level optimal $Q$ function—be $L$-smooth. Then for all $\mathfrak{b}$, $|V^{\ast}(\mathfrak{b})-V^{\times k\ast}(\mathfrak{b})|\leq\frac{2kL}{1-\gamma}$, where $V^{\times k\ast}(\mathfrak{b})$ is the value of state $\mathfrak{b}$ under an optimal $k$-aggregate policy.777The reparameterized problem is also an MDP, so the optimal value function and deterministic policy are well-defined. *

This theorem is proved by constructing a policy which switches actions every $k$ events and showing that it has bounded regret. This policy, at the start of any $k$-event period, adopts the optimal action from the unaggregated MDP at the initiating state. Due to smoothness, Q-values cannot drift by more than $kL$ during the period, after which the policy corrects itself. This, together with the reasoning abouve, offers an amplification guarantee:

Theorem 2**.**

*In an $L$-smooth MDP, let $k$ be a fixed repetition horizon. For any belief state where $A(\mathfrak{b})\geq 2kL$, the $k$-aggregate-horizon advantage is bounded below: *

[TABLE]

This result is especially useful when the event-level advantage is more than $\sigma=\frac{2kL}{1-\gamma}$. In this case, an aggregation horizon of $k$ can mitigate the adverse effects of approximating belief state with an $\varepsilon$-sufficient statistic for for an $\varepsilon$ up to:

[TABLE]

at the cost of the aggregation loss of $\frac{2kL}{1-\gamma}$.

Figs. 2(c) and  2(d) illustrate the benefit of action aggregation: they show the Q-values of the $k$-aggregated CK model with $k=5$ with both perfect and imperfect state estimation, respectively (the amount of noise is the same as inFig. 2(b)). As we show Sec. 4.4, the recovered policies incur very little loss due to state estimation error.

We conclude with the following observation.

Corollary 1**.**

Optimal repeating policies are near-optimal for the event-level problem as $L\rightarrow 0$ and amplification at every state is guaranteed.

4.3 Temporal Regularization: Switching Cost

As discussed above, temporal aggregation is guaranteed to improve learning in slow environments. It has, however, certain practical drawbacks due to its inflexibility. One such drawback is that, in the non-Markovian setting induced by belief-state approximation, training data should ideally be collected using a $k$-aggregated behavior policy.888This is unnecessary if the system is Markovian, since $(s,a,r,s^{\prime})$ tuples may be reordered to emulate any behavioral policy. Another drawback arises if the $L$-smoothness assumption is partially violated. For example, if certain rare events cause large changes in state or reward for short periods, the changes in Q-values may be abrupt, but are harmless from an SNR perspective if they induce large advantage gaps. An agent “committed” to a constant action during an aggregation period is unable to react to such events. We thus propose a more flexible advantage amplification mechanism, namely, a switching-cost regularizer. Intuitively, instead of fixing an aggregation horizon, we impose a fictitious cost (or penalty) $T$ on the agent whenever it changes its action.

More formally, the goal in the switching-cost (belief-state) MDP is to find an optimal policy defined as:

[TABLE]

This problem is Markovian in the extended state space $\mathcal{B}\times A$ representing the current (belief) state and the previously executed action. This state space allows the switching penalty to be incorporated into the reward function as $R(\mathfrak{b},a_{t-1},a_{t})=R(\mathfrak{b},a_{t})-T\cdot{\mathbbm{1}}[a_{t}\neq a_{t-1}]$.

The switching cost induces an implicit adaptive action aggregation—after executing action $a$, the agent will keep repeating $a$ until the cumulative advantage of switching to a different action exceeds the switching cost $T$. We can use this insight to bound the maximum regret of such a policy (relative to the optimal event-level policy) and also provide an amplification guarantee, as in the case with action aggregation.

In the case of problems with $2$ actions, we can analyze the action of the switching cost regularizer in a relatively intuitive way. As with Thm. 1, we bound the regret induced by the switching cost by constructing a policy that behaves as if it had to pay $T$ with every action switch. In particular, the optimal policy under this penalty adopts the action of the event-level optimal policy at some state $\mathfrak{b}_{t}$, then holds it fixed until its expected regret for not switching to a different action dictated by the event-level optimal policy exceeds $T$. Suppose the time at which this switch occurs is ($t+\omega$). The regret of this agent is no more than the regret of an agent with the option of paying $T$ upfront in order to follow the event-level optimal policy for $\omega$ steps. We can show that the same regret bound holds if the agent were paying to switch to the best fixed action for $\omega$ steps instead of following the event-level optimal policy. This allows derivation of the following bound:

Theorem 3**.**

*The regret of the optimal switching cost policy for a $2$-action MDP is less than $\frac{2\kappa L}{1-\gamma}$, where *

[TABLE]

and where $W$ is the Lambert W-function Corless et al. (1996).

This leads to an amplification result, analogous to Thm. 2:

Theorem 4**.**

Let $\kappa$ be as in Thm. 3. Any state whose advantage in the event-level optimal policy is at least $(1+\frac{1}{1-\gamma})2\kappa L$ has an advantage of at least $2T$ in the switching-cost regularized optimal policy.

4.4 Empirical Illustration

We experiment with synthetic models to demonstrate the theoretical results above. In a first experiment, we apply both action aggregation and switching cost regularization to the simple Choc-Kale POMDP, with parameters $\beta=0.9,\tau=0.25,\mu_{\mathrm{Choc}}=8,\mu_{\mathrm{Kale}}=2$, and $\sigma_{\mathrm{Choc}}=\sigma_{\mathrm{Kale}}=0.5$. To illustrate the effects of faulty state estimation, we corrupt the satisfaction level $s$ with noise drawn from a Gaussian (mean [math], stdev. $\sigma_{N}$), truncated on $[-1,1]$. As we increase $\sigma_{N}$, state estimation becomes worse. To mitigate this effect, we apply aggregations of $3,5$ actions at discounts of $\gamma=0.95$ and $0.99$ (Fig. 3a,b) and switching costs of $1,2,3$ (Fig. 3c). For each parameter setting, we train $10$ tabular policies with $Q$-learning, discretizing state space into $50$ buckets. For each training run, we roll-out $30000$ event-level transitions, exploring using actions taken uniformly at random—aggregated actions in the aggregation setting—then evaluate the discounted return of each policy using $100$ Monte Carlo rollouts of length $1000$. Figs.3a, b and c show the average performance across the $10$ training runs (with the 95% confidence interval) as a function of the $\sigma_{N}$. We see that action aggregation has a profound effect on solution quality, improving the performance of the policy up to a factor of almost $2$ (for $\gamma=0.99$). Switching cost regularization has a more subtle effect, providing more modest improvements in performance. We observe that over-regularized policies actually perform worse than the unregularized policy. We conjecture that this stark difference in performance is due to action aggregation having a double effect on the value estimates—apart from amplification, it also provides a more favorable behavioral policy.

A second experiment takes a more “options-oriented” perspective on the problem. Here, recommendable items have a continuous “kaleness” score between 0 and 1, with item $i$’s score denoted $v(i)$. At each time step, a set of $7$ items is drawn from a $[0,1]$-truncated Gaussian with mean equal to the kaleness score of the previously consumed item. The RL agent sets a target kaleness score $\theta\in\{0,0.25,0.5,0.75,1\}$ (its action space). This translates to a specific “presentation” of the $7$ items to the user such that the user is nudged to consume an item whose score is closer to the target. Specifically, the user chooses an item $i$ using a softmax distribution: $P(i)\propto\exp(-|v(i)-\theta|/\lambda)$, with temperature $\lambda=0.2$. The results are shown in Fig. 3d and exhibit a comparable level of improvement as in the binary-action case.

5 Related Work

The study of time series at different scales of granularity has a long-standing history in econometrics, where the main object of interest seems to be the behavior of various autoregressive models under aggregation Silvestrini and Veredas (2008), however, the behavior of aggregated systems under control does not seem to have been investigated in that field.

In RL, time granularity arises in several contexts. Classical semi-MDP/options theory employs temporal aggregation to organize the policy space into a hierarchy, where a pre-specified sub-policy, or option is executed for some period of time (termination is generally part of the option specification) Sutton et al. (1999). That options might help with partial observability (“state noise”) has been suggested—e.g., Daniel et al. Daniel et al. (2016), who also informally suggest that reduced control frequency can improve SNR; however the task of characterizing this phenomenon formally has not been addressed to the best of our knowledge. The learning to repeat framework (see Sharma et al. (2017) and references therein) provide a modeling perspective that allows an agent to choose an action-repetition granularity as part of the action space itself, but does not study these models theoretically. SNR has played a role in RL, but in different ways than studied here, e.g., as applied to policy gradient (rather than as a property of the domain) Roberts and Tedrake (2009).

The effect of the advantage magnitude (also called action gap) on the quality and convergence of reinforcement learning algorithms was first studied by Farahmand fahramand2011. Bellemare et al. bellemare2015 observed that the action gap can be manipulated to improve the quality of learned polices by introducing local policy consistency constraints to the Bellman operator. Their considerations are, however, not bound to specific environment properties.

Finally, our framework is closely related with the study of regularization in RL and its benefits when dealing with POMDPs. Typically, an entropy-based penalty (or KL-divergence w.r.t. to a behavioral policy) is added to the reward to induce a stochastic policy. This is usually justified in one of several ways: inducing exploration Nachum et al. (2017); accelerating optimization by making improvements monotone Schulman et al. (2015); and smoothing the Bellman equation and improving sample efficiency Fox et al. (2016). Of special relevance is the work of Thodoroff et al. Thodoroff et al. (2018), who, akin to this work, exploit the sequential dependence of Q-values for better Q-value estimation. In all this work, however, regularization is simply a price to pay to achieve a side goal (e.g., better optimization/statistical efficiency). While stochastic policies often perform better than deterministic ones when state estimation is deficient Singh et al. (1994), and methods that exploit this have been proposed in restricting settings (e.g., corrupted rewards Everitt et al. (2017)), the connection to regularization has not been made explicitly to the best of our knowledge.

6 Concluding Remarks

We have developed a framework for studying the impact of belief-state approximation in latent-state RL problems, especially suited to slowly evolving, highly noisy (low SNR) domains like recommender systems. We introduced advantage amplification and proposed and analyzed two conceptually simple realizations of it. Empirical study on a stylized domain demonstrated the tradeoffs and gains they might offer.

There are a variety of interesting avenues suggested by this work: (i) the study of soft-policy regularization for amplification (preliminary results are presented in the long version of this paper); (ii) developing techniques for constructing more general “options” (beyond aggregation) for amplification; (iii) developing amplification methods for arbitrary sources of modeling error; (iv) conducting more extensive empirical analysis on real-world domains.

Appendix A Technical Results and Additional Material

A.1 Analysis of Action Aggregation

We start with an observation that frames the subsequent analysis.

Lemma 1** (Counterfactual Q-values).**

Let $\pi$ and $\rho$ be two deterministic policies and let the $Q$-function of $\pi$, $Q^{\pi}$ be known. Then, for any state $\mathfrak{b}\in\mathcal{B}$:

[TABLE]

where $\operatorname*{\mathbb{E}}_{\rho}$ the expectation is over trajectories $(\mathfrak{b}_{i})_{i\in\mathbb{N}}$ generated by $\rho$ starting at $\mathfrak{b}$ ($\mathfrak{b}_{0}=\mathfrak{b}$).

Proof.

Let $\rho\pi$ be a non-stationary policy that starting at $\mathfrak{b}$ executes $\rho(\mathfrak{b})$ exactly once, then follows $\pi$ forever. Then,

[TABLE]

Linearity of expectation and the boundedness of both $Q$ functions ensures that recursive application to $Q^{\pi}(\mathfrak{b}^{\prime},\pi(\mathfrak{b}^{\prime}))-Q^{\rho}(\mathfrak{b}^{\prime},\rho(\mathfrak{b}^{\prime}))$ converges to the desired quantity. ∎

This allows us to measure the behavior of $k$-aggregate policies (in terms of advantages) relative to atomic ones.

Lemma 2**.**

Let $\mathfrak{b}$ be a state for which the optimal action $a$ does not change within $k$ steps. In other words, for all $\mathfrak{b}^{\prime}\in{\cal B}^{\times k}(\mathfrak{b})$, $\pi^{*}(\mathfrak{b}^{\prime})=a$, where ${\cal B}^{\times k}(\mathfrak{b})$ denotes all states reachable from $\mathfrak{b}$ in $k$ steps under some action sequence. Then for any other action $a^{\prime}$,

[TABLE]

where the expectation is over the trajectory $\mathfrak{b}^{a^{\prime}}_{1},\ldots,\mathfrak{b}^{a^{\prime}}_{k}$ of states that follow after $\mathfrak{b}$ when taking action ${a^{\prime}}$ and $\mathfrak{b}^{a^{\prime}}_{0}=\mathfrak{b}$ for notational convenience.

Proof.

Let $\pi$ be a policy that executes $a$ for $k$ steps and then reverts to the optimal policy and $\rho$ be a policy that executes $b$ for $k$ steps and then reverts to the optimal policy. We apply Lemma 1 noting that $\pi$ coincides with the optimal policy, hence $Q^{\pi}=Q^{\ast}$. ∎

Lemma 2 establishes that advantage of $k$-aggregate actions is the compound discounted advantage of the atomic ones. This, combined with the smoothness of the optimal $Q$-function allows us to analyze the effects of action aggregation. In particular, suppose that we have found a state $\mathfrak{b}$ such that $A(\mathfrak{b})=Q^{\ast}(\mathfrak{b},a)-\max_{a^{\prime}\neq a}Q^{\ast}(\mathfrak{b},a^{\prime})\geq 2kL$. The smoothness of the $Q$-function allows us to infer that for any action taken, the advantage at the next state will be at least $2kL-2L$, $2kL-4L$ after $2$ steps and so on (this guarantees the advantage gap can’t close in less than $k$ steps). Hence if we were to replace atomic actions with $k$-repeated actions at states with advantage of more than $2kL$, the following can be observed:

Lemma 3**.**

Consider an MDP with an $L$-smooth optimal $Q$-function $Q^{\ast}$, and the reparametrization by holding actions fixed for $k$ steps whenever the advantage gap is greater than $2kL$. Then, at each state, either:

[TABLE]

or $A(\mathfrak{b})\leq 2kL$.

Proof.

As established, $A^{\times k}(\mathfrak{b})=E[\sum_{i=0}^{k}\gamma^{i}A(\mathfrak{b}_{i}^{a^{\prime}})]$. Smoothness of the $Q$-function guarantees that $A(\mathfrak{b}_{i+1}^{a^{\prime}})\geq A(\mathfrak{b}_{i}^{a^{\prime}})-2L$, resp. $A(\mathfrak{b}_{k}^{a^{\prime}})\geq A(\mathfrak{b})-2kL$ as discussed above. Hence

[TABLE]

Here, the first expression comes from the finite sum geometric series formula, and the second term reflects the fact that

[TABLE]

∎

We have thus far considered the situation where aggregation is state-dependent, based on the knowledge of the advantage amplitude. A more realistic implementation of aggregation would be to fix some global static $k$ apriori. While the reasoning is similar to the above, there is an added complexity that the aggregation itself comes at a price – the values of certain states will be reduced (relative to the best atomic policy) due to the inability to rapidly switch actions. This additional cost must then be factored into the bounds.

We first bound the cost of the best slow policy under smooth $Q$ assumptions.

Theorem 1.

Let $k$ be a fixed horizon and $Q^{\ast}$, the event-level optimal Q-function be $L$-smooth. Then for all $\mathfrak{b}$, $|V^{\ast}(\mathfrak{b})-V^{\times k\ast}(\mathfrak{b})|\leq\frac{2kL}{1-\gamma}$, where $V^{\times k\ast}(\mathfrak{b})$ is the value of state $\mathfrak{b}$ under an optimal $k$-aggregate policy999Note that the reparametrized problem in which a decision can be made every $k$ atomic steps is also an MDP, so the notion of an optimal value function (one that provides the largest possible value at every state) resp. optimal deterministic policy is well-defined. .

Proof.

We lower bound the return of the optimal $k$-aggregate policy by exhibiting a (not-necessarily optimal) $k$-aggregate policy with sufficient returns. In particular, let $\pi$ be a deterministic optimal atomic policy, and $\rho$ be the policy that at state $\mathfrak{b}_{t}$ executes $\pi(\mathfrak{b}_{t})$ for the next $k$ steps, then executes $\pi(\mathfrak{b}_{t+k})$, etc. Following Lemma 1, we have that

[TABLE]

Let $(\mathfrak{b}_{ik})_{i\in\mathbb{N}}$ be the states at which $\rho$ may switch actions. At any $\mathfrak{b}_{ik}$, if $A(\mathfrak{b}_{ik})\geq 2kL$, due to smoothness of $Q^{*}$, we know that $a$ remains optimal for the next $k$ steps thus $\rho(\mathfrak{b}_{ik+j})=\pi(\mathfrak{b}_{ik+j})$ and $Q^{*}(\mathfrak{b}_{ik+j},\pi(\mathfrak{b}_{ik+j}))-Q^{*}(\mathfrak{b}_{ik+j},\rho(\mathfrak{b}_{ik+j}))=0$ until the next decision point for $\rho$ (i.e. for $j\leq k-1$). Conversely, if $A(\mathfrak{b}_{ik})<2kL$, then in the worst case, assuming that the Q-value of the optimal action $Q^{*}(\mathfrak{b}_{ik+j},a)$ decreases by $L$ with every step $j$ and the Q-value of a suboptimal action $Q^{*}(\mathfrak{b}_{ik+j},a^{\prime})$ increases respectively by $L$, then for any of the next $k$ steps, $Q^{*}(\mathfrak{b}_{ik+j},a)-Q^{*}(\mathfrak{b}_{ik+j},a^{\prime})\geq-2kL$. Hence, for all $i$, $|Q^{*}(\mathfrak{b}_{i},\pi(a))-Q^{*}(\mathfrak{b}_{i},\rho(a)))|$ is either [math] or less than $2kL$, leading to

[TABLE]

∎

We now factor in the aggregation loss from Thm. 1 into Lemma 3, which together characterize the amplification properties of action aggregation.

Theorem 2.

In an $L$ smooth MDP, let $k$ be a fixed repetition horizon. For any state where the advantage gap $A(\mathfrak{b})$ greater than $2kL$, the fixed-horizon advantage is lower-bounded as follows:

[TABLE]

Proof.

Let $\bar{Q}^{\times k\ast}$ be the losslessly amplified $Q$-function as in Lemma 3. Since $A(\mathfrak{b})\geq 2kL$, action $a$ is optimal under an atomic optimal policy for the next $k$ periods and due to Thm. 1,

[TABLE]

Moreover, for any $a^{\prime}\neq a$

[TABLE]

as the expected return following the aggregation period of $k$ can only be lower than the one of the losslessly amplified problem. Thus:

[TABLE]

The bound from the expectation term comes from Lemma 3. ∎

A.2 Analysis of Switching Cost

Let $Q^{*}$ be an L-smooth optimal Q-function of a POMDP with an optimal deterministic policy $\pi^{*}$ and $T$ be a switching cost. Consider the following policy $\rho$: at $\mathfrak{b}_{0}$, $\rho$ adopts the optimal atomic action $a_{0}=\rho_{0}(\mathfrak{b}_{0})=\pi^{*}(\mathfrak{b}_{0})$. Also at $t=0$, $\rho$ calculates the time until its regret for repeating $a_{0}$ relative to following the optimal policy exceeds, $T$, i.e.

[TABLE]

(where the expectation is, as before, over trajectories of belief sates generated by executing $a_{0}$, conditioned on the realization of $\mathfrak{b}_{0}$) and then repeats $a_{0}$ until $t=\omega_{0}$. At $t=\omega_{0}$, $\rho$ queries the optimal policy for $a_{\omega_{0}}=\pi^{*}(\mathfrak{b}_{\omega_{0}})$, executes it until time $\omega_{1}$ and so on. In other words, $\rho$ repeats an action adopted from the optimal policy until its regret for not having followed the optimal policy exceeds $T$ and then switches.

The return of $\rho$ is equivalent to the return of a policy that would pay $T$ upfront to switch to and follow the optimal policy for $\omega$ steps. We first bound the loss of this policy and then argue that it also upper-bounds the loss of the optimal switching cost policy.

Lemma 4**.**

The total regret of $\rho$ relative to $\pi^{*}$ is no more than

[TABLE]

Proof.

Starting from Lemma 1, the total loss of $\rho$ relative to $\pi^{*}$ is

[TABLE]

We now argue that $\mathbb{E}_{\mathfrak{b}_{i}\sim\rho}\left[Q^{*}(\mathfrak{b}_{i},\pi^{*}(\mathfrak{b}_{i}))-Q^{*}(\mathfrak{b}_{i},\rho_{i})\right]$ is bounded at any step $i$. Observe that the action taken by the policy at time $i$ depends only on the belief state realization $\mathfrak{b}_{\omega_{-1}}$, $\omega_{-1}$ being the most recent time point at which the policy was allowed to switch actions. Thus, by the law of total probability,

[TABLE]

In the above, we have just rewritten the loss relative to the last switching point $\omega_{-1}$. We now show that the conditional expectation $\mathbb{E}_{\mathfrak{b}_{i}\sim\rho}\left[Q^{*}(\mathfrak{b}_{i},\pi^{*}(\mathfrak{b}_{i}))-Q^{*}(\mathfrak{b}_{i},\rho_{i})|\mathfrak{b}_{\omega_{-1}}\right]$ is bounded for any realization $\mathfrak{b}_{\omega_{-1}}$. Recall that at time $\omega_{-1}$, $\rho$ picks the action $a_{\omega{-1}}=\pi^{*}(\omega_{-1})$ and executes it until time $\omega$, the time when its expected regret exceeds $T$, and then switches. Thus, the highest achievable per-event expected regret conditioned on $\mathfrak{b}_{\omega_{-1}}$ occurs when at time $\omega_{-1}$, there exists an alternative action $a^{\prime}\neq a_{\omega{-1}}$ having the same Q-value, i.e. $Q^{*}(\mathfrak{b}_{\omega_{-1}},a_{\omega_{-1}})=Q^{*}(\mathfrak{b}_{\omega_{-1}},a^{\prime})$, and then, in subsequent time steps, the expected $Q$-value of $a^{\prime}$, $\mathbb{E}_{\mathfrak{b}_{i}\sim\rho}\left[Q^{*}(\mathfrak{b}_{i},a^{\prime})|\mathfrak{b}_{\omega_{-1}}\right]$, starts increasing at the maximum rate of $L$ while the Q-value of $a_{\omega_{-1}}$ starts decreasing at the same rate. The maximum rate of increase of the expectation is justified since if the Q-value of $a^{\prime}$ is $L$-smooth over individual sample paths, i.e. $|Q^{*}(\mathfrak{b}_{i},a)-Q^{*}(\mathfrak{b}_{i+1},a)|\leq L$, the sequence of expectations is also $L$-smooth, $|\mathbb{E}_{\mathfrak{b}_{i}\sim\rho}\left[Q^{*}(\mathfrak{b}_{i},a)|\mathfrak{b}_{\omega_{-1}}\right]-\mathbb{E}_{\mathfrak{b}_{i+1}|\mathfrak{b}_{\omega_{-1}}\sim\rho}\left[Q^{*}(\mathfrak{b}_{i+1},a)|\mathfrak{b}_{\omega_{-1}}\right]|\leq L$). By the same reasoning as in Lemma 3 (but this time for the sequence of expectations, rather than the sequence of realizations), the regret for not following $\pi^{*}$ after $k$ steps is bounded by $2\gamma L\frac{1+k\gamma^{k}-(1+k)\gamma^{k}}{(1-\gamma)^{2}}$. Let us calculate the minimum $k$ (under assumptions of monotone regret increase at the maximum rate) for which this regret exceeds $T$ as $\lceil\kappa\rceil$, where $\kappa$ is the solution of $2\gamma L\frac{1+k\gamma^{k}-(1+k)\gamma^{k}}{(1-\gamma)^{2}}=T$.

[TABLE]

where $W$ is the Lambert W-function. Knowing $\kappa$, we can deduce that the maximum per-event expected regret, $\mathbb{E}_{s_{i}\sim\rho}\left[Q^{*}(s_{i},\pi^{*}(s_{i}))-Q^{*}(s_{i},\rho_{i})|\mathfrak{b}_{\omega_{-1}}\right]$ is no more than

[TABLE]

under worst-case assumptions. This leads to an overall regret bound of $\epsilon/(1-\gamma)$. ∎

So far, we have produced an agent that pays a cost $T$ to switch to the optimal policy for some time-specific horizon $\omega_{t}$ and bounded its regret. This is not quite what we need, since $\rho$ gets more than one action switch for free within the horizon over which it is allowed to follow the optimal policy. We now argue that for a slow environment, this bound also holds for an agent which pays on every switch. To do so, we need to produce a reasonable policy that pays for every action switch.

Let $A(\mathfrak{b},\pi^{*},a^{\times k})$ be the advantage of following the optimal policy for $k$ turns over repeating $a$ for the same time and then switching to the optimal policy (resp. the regret for repeating $a$ over following the optimal policy). That is, following Lemma 1,

[TABLE]

where the expectation is over trajectories generated by taking action $a$ and $\mathfrak{b}_{0}=\mathfrak{b}$.

Furthermore, let $\omega(\mathfrak{b})=\operatorname*{arg\,min}_{k}\min_{a^{\prime}\in A}k$ s.t. $A(\mathfrak{b},\pi^{\ast},a_{0}^{\times k})-A(\mathfrak{b},\pi^{\ast},a^{\prime\times k})\geq T$ if the former is feasible, else $\infty$. In words, $\omega(\mathfrak{b})$ is the shortest time until the regret of having repeated the current action $a_{0}$ exceeds the regret of having repeated some other action $a^{\prime}$ by $T$. Note that this does not need to be finite, since in some state, repeating any action for any amount of time might yield similar returns.

Based on this, we construct a policy $\sigma$ that behaves as if it would pay $T$ on every switch. At $\mathfrak{b}_{0}$, $\sigma$ executes the atomic optimal action $\pi^{*}(\mathfrak{b}_{0})$. At the next state, $\mathfrak{b}_{1}$, if $\omega(\mathfrak{b}_{1})$ is infinite, $\sigma$ repeats $a_{0}$ once. Otherwise, $\sigma$ repeats $a_{0}$ $\omega(\mathfrak{b}_{1})$ times and then switches to $a^{\prime}$, the action whose regret exceeds $T$ after $\omega(\mathfrak{b}_{1})$ steps. This policy can do no better than a policy that pays $T$ upfront to switch to $a^{\prime}$ and maintain it for $\omega(\mathfrak{b}_{1})$ steps. By extension, $\sigma$ can generate no more return than the best switching cost policy. Now we can bound its loss.

Theorem 3.

The regret of the optimal switching cost policy for a $2$-action MDP is less than $\frac{2\kappa L}{1-\gamma}$, where $\kappa$ is the same as in the previous theorem.

Proof.

Again we must scrutinize the sequence $\left(\mathbb{E}_{\mathfrak{b}_{i}\sim\sigma}\left[Q^{*}(\mathfrak{b}_{i},\pi^{*}(\mathfrak{b}_{i}))-Q^{*}(\mathfrak{b}_{i},\sigma_{i})\right]\right)_{i\in\mathbb{N}}$ showing that it is bounded at any step $i$. We consider the contrapositive: suppose there existed $i$, such that

[TABLE]

with $\kappa$ as in the previous theorem. By the law of total probability,

[TABLE]

The expectation over $\mathfrak{b}_{i-\kappa}$ in the second line above implies that for at least one possible realization of $\mathfrak{b}_{i-\kappa}$, the expected Q difference after $\kappa$ periods exceeds $2\kappa L$, i.e. $\mathbb{E}_{\mathfrak{b}_{i}\sim\sigma}\left[Q^{*}(\mathfrak{b}_{i},\pi^{*}(\mathfrak{b}_{i}))-Q^{*}(\mathfrak{b}_{i},\sigma_{i})|\mathfrak{b}_{i-\kappa}\right]\geq 2\kappa L$. Let $\hat{\mathfrak{b}}$ be such a realization of $\mathfrak{b}_{i-\kappa}$ and let $a^{*}=\pi^{*}(\mathfrak{b}_{i})$ resp. $a=\sigma(\mathfrak{b}_{i})$ . Due to the $L$-smoothness of the sequences of expected Q-values $\left(\mathbb{E}_{\mathfrak{b}_{j}\sim\rho}\left[Q^{*}(\mathfrak{b}_{j},a^{*})|\mathfrak{b}_{i-\kappa}=\hat{\mathfrak{b}}\right]\right)_{j\in[i-\kappa,\ldots,i]}$, $\left(\mathbb{E}_{\mathfrak{b}_{j}\sim\rho}\left[Q^{*}(\mathfrak{b}_{j},a)|\mathfrak{b}_{i-\kappa}=\hat{\mathfrak{b}}\right]\right)_{j\in[i-\kappa,\ldots,i]}$, we know that for all ${j\in[i-\kappa,\ldots,i]}$, $\mathbb{E}_{\mathfrak{b}_{j}\sim\rho}\left[Q^{*}(\mathfrak{b}_{j},a^{*})|\mathfrak{b}_{i-\kappa}=\hat{\mathfrak{b}}\right]\geq\mathbb{E}_{\mathfrak{b}_{j}\sim\rho}\left[Q^{*}(\mathfrak{b}_{j},a)|\mathfrak{b}_{i-\kappa}=\hat{\mathfrak{b}}\right]$. It is straightforward to verify that for any $L$-smooth sequences of real numbers $(A_{i})_{i\in[1,\ldots,k]}\geq(B_{i})_{i\in[1,\ldots,k]}$ of length $\kappa$, such that $A_{\kappa}-B_{\kappa}>2\kappa L$, the discounted sum $\sum_{i=1}^{\kappa}\gamma^{i-1}(A_{i}-B_{i})$ must be greater than $T$. Hence, the cumulative expected regret of taking $a^{*}$ vs. $a$ following state $\hat{\mathfrak{b}}$ must exceed $T$ and $\sigma$ must have switched by time $i$, which is a contradiction with the assumption that $\sigma$ executes $a$.

∎