A Concave Value Function Extension for the Dynamic Programming Approach to Revenue Management in Attended Home Delivery

TL;DR

This paper extends dynamic programming methods for revenue management in attended home delivery by establishing the concavity and fixed point properties of the value function, enabling scalable approximation techniques.

Contribution

It introduces a novel concave extension of the value function in dynamic programming for revenue management, facilitating scalable approximate solutions.

Findings

The value function admits a continuous, concave extension under certain conditions.

The dynamic programming operator has a unique fixed point.

Numerical examples illustrate the practical applicability of the theoretical results.

Abstract

We study the approximate dynamic programming approach to revenue management in the context of attended home delivery. We draw on results from dynamic programming theory for Markov decision problems, convex optimisation and discrete convex analysis to show that the underlying dynamic programming operator has a unique fixed point. Moreover, we also show that -- under certain assumptions -- for all time steps in the dynamic program, the value function admits a continuous extension, which is a finite-valued, concave function of its state variables. This result opens the road for achieving scalable implementations of the proposed formulation, as it allows making informed choices of basis functions in an approximate dynamic programming context. We illustrate our findings using a simple numerical example and conclude with suggestions on how our results can be exploited in future work to obtain closer approximations of the value function.

Tables

Table 1. Table 1: The parameters of the numerical example.

$\lambda$	$0.5$
$\overline{t}$	$200$
$[\underset{¯}{d},\overline{d}]$	$[0,2]$
$r$	1
$S$	$\{1,2\}$
$\overline{x}$	$[4,4]$
${\beta }_c$	$1$
${\beta }_d$	$-1$
${\beta }_1$	$1$
${\beta }_2$	$-1$
$C_{{ℝ}_{+}}\left(x\right)$	$2+x_1+2x_2$

Equations

$$C(x):=\{\begin{array}{ll}{C}_{{\mathbb{R}}_{+}}(x)& x\in X\\ \infty & \text{otherwise,}\end{array}$$

$${\Pi }_{a,s}(d):=\frac{\exp ({\beta }_c+{\beta }_s+{\beta }_d{d}_{a,s})}{{\sum }_{k\in S}\exp ({\beta }_c+{\beta }_k+{\beta }_d{d}_{a,k})+1},$$

$$V_t(x):=$$

$$\text{ where }{V}_{T+1}(x)=-C(x)\forall x\in X,$$

$$V_{t-1}:=\mathcal{T}{V}_t.$$

$${\mathcal{V}}_X:=\left\{v\in {\mathbb{R}}_{+}^N|\sum _{i\in N}{v}_i=1,N=\dim (X)\right\}.$$

$$\tilde{f}(x):=\inf \left\{a^{⊺}x+b|a^{⊺}y+b\ge f(y)\right\}$$

$$f(x)\ge \sum _{q\in Q}{\mu }_{q}f(q).$$

$$g_t(x,p):=\sum _{a,s}{p}_{a,s}\left[V_t(x+1_{a,s})-V_t(x)\right]+V_t(x)$$

$$\begin{array}{rl}& \sum _{a,s}{p}_{a,s}\left[V_t(x+1_{a,s})-V_t(x)\right]+{ϵ}_t(x,Q)\\ \ge & \sum _{q\in Q}\left\{\sum _{a,s}{q}_{a,s}^{(p)}\left[V_t(q^{(x)}+1_{a,s})-V_t(q^{(x)})\right]+V_t(q^{(x)})\right\},\end{array}$$

$$\begin{array}{rl}\sum _{a,s}{p}_{a,s}{w}_t+{ϵ}_t(x,Q)\ge & \sum _{q\in Q}{\mu }_q\sum _{a,s}{q}_{a,s}^{(p)}{W}_t\\ ⟺\lambda \le & \frac{{ϵ}_t(x,Q)}{(W_t-w_t){\sum }_{a,s}\pi (a){\Pi }_{a,s}}.\end{array}$$

$$\begin{array}{rl}& p\left[V_t(x+1)-V_t(x)\right]+V_t(x)\\ & \ge \sum _{q\in Q}{\mu }_q\left\{p\left[V_t\left(q^{(x)}+1\right)-V_t\left(q^{(x)}\right)\right]+V_t\left(q^{(x)}\right)\right\}.\end{array}$$

$${\tilde{V}}_t(x+p)\ge \sum _{q\in Q}{\mu }_q{\tilde{V}}_t\left(q^{(x)}+q^{(p)}\right).$$

$$V^{\ast }(x):=(\bar{d}+r)1^{⊺}(\bar{x}-x)-C(\bar{x})\text{ for all }x\in X.$$

$$V_{t-1}(x):=\underset{d}{\max }\sum _{y\in X}{P}_{x,y}(d)[\tilde{g}(x,d,y)+V_t(y)],$$

$$V_{t-1}(x)=\underset{d}{\max }\left\{g(x,d)+\sum _{y\in X}{P}_{x,y}(d)V_t(y)\right\}.$$

$$v^{\ast }(x):=\{\begin{array}{ll}1+\underset{d}{\max }\left\{\sum _{y\in X}{P}_{x,y}(d)v^{\ast }(y)\right\}& x\in X\backslash \{\bar{x}\}\\ 1& x=\bar{x}.\end{array}$$

$$\rho$$

$$=\underset{x\in X}{\max }\left\{\frac{\underset{d\in D}{\max }{\sum }_{y\in X}{P}_{x,y}(d)v^{\ast }(y)}{v^{\ast }(x)}\right\}.$$

$$\Vert V_t\Vert :=\underset{x\in X}{\max }\left\{\frac{|V_t(x)|}{v^{\ast }(x)}\right\}$$

$$\Vert \mathcal{T}{V}_t-\mathcal{T}{V}_{t^{\prime }}\Vert \le \rho \Vert V_t-V_{t^{\prime }}\Vert$$

$$0$$

$$0=\underset{d}{\max }\left\{\sum _{a,s}{p}_{a,s}(d)[d_{a,s}-\bar{d}]\right\}.$$

$$\frac{p_{a,s}}{p_{a,0}}=\exp ({\beta }_c+{\beta }_s+{\beta }_d{d}_{a,s}),$$

$$d_{a,s}={\beta }_d^{-1}\left[\ln \left(\frac{p_{a,s}}{p_{a,0}}\right)-{\beta }_c-{\beta }_s\right].$$

$$\begin{array}{rl}& \begin{array}{rl}\mathcal{T}{V}_t(x)& =\underset{p}{\max }\sum _{a,s}{p}_{a,s}\{r+{\beta }_d^{-1}\left[\ln \left(\frac{p_{a,s}}{p_{a,0}}\right)-{\beta }_c-{\beta }_s\right]\\ & +V_t(x+1_{a,s})-V_t(x)\}+V_t(x)\end{array}\\ & =\underset{p}{\max }\{f(p)+g_t(x,p)\},\end{array}$$

$$f(p):=\sum _{a,s}{p}_{a,s}\left\{r+{\beta }_d^{-1}\left[\ln \left(\frac{p_{a,s}}{p_{a,0}}\right)-{\beta }_c-{\beta }_s\right]\right\}$$

$$\Vert \mathcal{T}{V}_t-\mathcal{T}{V}_{t^{\prime }}\Vert \le \rho \Vert V_t-V_{t^{\prime }}\Vert .$$

$$\Vert {\mathcal{T}}^N{V}^{\ast }-{\mathcal{T}}^N{V}_{\bar{t}+1}\Vert$$

$$\Vert V^{\ast }-V_{\bar{t}+1-N}\Vert$$

$$\Vert \mathcal{T}{V}^{\ast }-\mathcal{T}{V}^{\ast \ast }\Vert \le \rho \Vert V^{\ast }-V^{\ast \ast }\Vert .$$

$$\begin{array}{rl}\underset{N\to \infty }{\lim }\Vert (\mathcal{T}{)}^N{V}^{\ast }-(\mathcal{T}{)}^N{V}^{\ast \ast }\Vert & \le \underset{N\to \infty }{\lim }{\rho }^N\Vert V^{\ast }-V^{\ast \ast }\Vert \\ ⟺\Vert V^{\ast }-V^{\ast \ast }\Vert & \le \underset{N\to \infty }{\lim }{\rho }^N\Vert V^{\ast }-V^{\ast \ast }\Vert \\ ⟹\Vert V^{\ast }-V^{\ast \ast }\Vert & \le 0.\end{array}$$

Full text

1 Introduction

1.1 Revenue Management and Attended Home Delivery

The expenditure of US households on online grocery shopping could reach 100 billion in 2022 according to the Food Marketing Institute [[1](#bib.bib1)]. Although growth forecasts vary and more conservative estimates lie, for example, at 30 billion for the year 2021 [2], the overall trend is clear: The online grocery sector is likely to grow if some of its main challenges can be overcome.

One of these challenges is managing the logistics as one of the main cost-drivers. In particular, one can seek to exploit the flexibility of customers by offering delivery options at different prices to create delivery schedules that can be executed in a cost-efficient manner. There are a number of ways to achieve this. Recent proposals include, for example, giving customers the choice between narrow delivery time windows for high prices and vice versa [3] or charging customers different prices based on the area and their preferred delivery time [4, 5, 6].

In this paper, we focus on the latter. We refer to the problem of finding the profit-maximising delivery slot prices as the revenue management problem in attended home delivery, where “attended” refers to the requirement that customers need to be present upon delivery of the typically perishable goods, which is in contrast to, for example, standard mail delivery. Note that attended home delivery problems are more complex than standard delivery services, because goods need to be delivered in time windows that were pre-agreed with the customers.

We adopt a dynamic programming (DP) model of an expected profit-to-go function, the value function of the DP, given the current state of orders and time left for customers to book a delivery slot. This DP was initially devised in the fashion industry [7], but subsequently adopted and refined by the transportation sector and the attended home delivery industry [5].

To find the optimal delivery slot prices, we need to compute the value function (at least approximately) for all states and times. The main challenge with this is that the state space of the DP grows exponentially with the set of delivery time slots, i.e. it suffers from the “curse of dimensionality”. This means that for industry-sized problems, the value function cannot be computed exactly, even off-line, because there are too many states. Our ultimate objective is to compute improved value function approximations. Therefore, we study in this paper how the value function of the exact DP behaves mathematically in time and across state variables.

In this paper we show that – under certain assumptions – for all time steps in the dynamic program, the value function admits a continuous extension, which is a finite-valued, concave function of its state variables. This result opens the road for achieving scalable implementations of the proposed formulation, as it becomes possible to make informed choices of basis functions in an approximate dynamic programming context.

Improved value function approximations could finally be used for calculating optimal delivery slot prices. This has been shown by [8], where it is proven that a unique set of optimal delivery slot prices exists, which can be found using simple Newton root search algorithms if estimates of the value function are known for all states and times.

Our paper is structured as follows: In the remainder of Section 1, we introduce some notation. Then in Section 2, we define the revenue management problem in attended home delivery and formulate it as a DP. In Section 3, we state the definitions and assumptions that our analysis is based on and present our main result, Theorem 2, which says that there exists a continuous extension of the value function of the exact DP that is a finite-valued, concave function in its state variables. Section 4 contains reformulations of the DP into mathematically more convenient forms and develops a series of supporting results leading to the proof of Theorem 2. Section 5 presents a numerical illustration of the proposed scheme, while Section 6 concludes the paper and suggests directions for future research.

1.2 Notation

Let $\mathbf{1}$ denote a column vector of ones. Given some $(a,s)$, let $1_{a,s}$ be a column vector build by stacking the transposed rows of a unit matrix with $1$ at the $(a,s)$th entry. Let $\mathbb{R}_{+}$ be the non-negative real numbers and let dim$(\cdot)$ denote the dimension of its argument. Let conv$(\cdot)$ denote the convex hull of its argument.

2 Revenue Management Problem Formulation

In this section, we derive a discrete-state formulation of the revenue management problem in attended home delivery.

2.1 Problem Statement

We model an online business that delivers goods to locations of known customers. We adopt a local approximation of the revenue management problem by dividing the service area geographically into a set of non-overlapping rectangular sub-areas $A:=\{1,2,\dots,\bar{a}\}$, where the customers in each area $a\in A$ are served by one delivery vehicle. This model resembles the setting in [6].

We consider a finite booking horizon with possibly unequally-spaced time steps indexed by $t\in T:=\{1,2,\dots,\bar{t}\,\}$. We refer to [5, Section 4.3] for details on how to obtain a customer arrivals model using a Poisson process with time-invariant event rate $\lambda\in(0,1)$ for all $t\in T$ from a Poisson process with homogeneous time steps, but time-varying event rate. The probability that a customer arrives from sub-area $a$ is given by $\pi(a)\in[0,1]$ with $\sum_{a\in A}\pi(a)=1$.

Customers can choose from a number of (typically 1-hour wide) delivery time windows, which we call slots $s\in S$, where $S:=\{1,2,\dots,\bar{s}\}$. Let $s=0$ correspond to a customer not choosing any slot. Each sub-area/delivery slot pair $(a,s)$ is assigned a delivery charge $d_{a,s}\in\left[\underline{d},\bar{d}\,\right]\cup\infty$, for some minimum allowable charge $\underline{d}\in\mathbb{R}$ (which is typically, though not necessarily, positive) and some maximum allowable charge $\bar{d}\geq\underline{d}$. The role of $d_{a,s}=\infty$ is a convention to indicate that slot $s$ is not offered in area $a$. This will be explained in more detail when introducing the customer choice model below.

We define $d:=\left\{d_{a,s}\ \left|\ (a,s)\in A\times S\right.\right\}$ consisting of the set of delivery charges that the business decides to charge at any time step $t\in T$, where it is to be understood that $d$ is a stacked vector including the different values that $d_{a,s}$ can take as $a$ and $s$ vary. Let the set of allowed decision vectors be $D:=\left\{d\ \left|\ d_{a,s}\in\left[\underline{d},\bar{d}\,\right]\text{ for all }(a,s)\in A\times S\right.\right\}$.

For each sub-area/delivery slot pair $(a,s)\in A\times S$, we denote the number of placed orders by $x:=\left\{x_{a,s}\ \left|\ (a,s)\in A\times S\right.\right\}\in\mathbb{Z}^{|A||S|}$. We define $X:=\left\{x\ \left|\ 0\leq x_{a,s}\leq\bar{x}_{a,s}\text{ for all }(a,s)\in A\times S\right.\right\}$, where $\bar{x}_{a,s}$ is a scalar indicating the maximum number of deliveries that can be fulfilled in the sub-area/delivery slot pair $(a,s)$. In general, we do not require the maximum number of deliveries to be the same for all areas and all slots, e.g. because this will depend on the size of the delivery area. Examples of computing this quantity can be found in [6, Section 4]. Let us also define $\bar{x}:=\left\{\bar{x}_{a,s}\ \left|\ (a,s)\in A\times S\right.\right\}$. Let $r\in\mathbb{R}$ denote the expected net revenue of an order, i.e. expected revenue minus costs prior to delivery. This is assumed to be invariant across all orders. We define

[TABLE]

where $C_{\mathbb{R}_{+}}:X\to\mathbb{R}_{+}$. The function $C$ approximates the delivery cost to fulfil the set of orders $x$. The precise delivery cost cannot be computed, as it is the solution to a vehicle routing problem with time windows, which is intractable for industry-sized applications [9].

Let the probability that a customer chooses sub-area/delivery slot pair $(a,s)$ if offered prices $d$ be $\Pi_{a,s}(d)$, such that $d\mapsto\Pi_{a,s}(d)\in[0,1)$ for all $(a,s)\in A\times S$. For all areas $a\in A$, note that $\sum_{s\in S}\Pi_{a,s}(d)=1-\Pi_{a,0}(d)$, where $\Pi_{a,0}$ denotes the probability of a customer from sub-area $a$ leaving the online ordering platform without choosing any delivery slot. A typical choice for $\Pi_{a,s}$ is the multinomial logit model that was also used in [6]:

[TABLE]

where $\beta_{c}\in\mathbb{R}$ denotes a constant offset, $\beta_{s}\in\mathbb{R}$ represents a measure of the popularity for all delivery slots and $\beta_{d}<0$ is a parameter for the price sensitivity. Note that the no-purchase utility is normalised to zero, i.e. for the no-purchase “slot” $s=0$, we have $\beta_{c}+\beta_{0}+\beta_{d}d_{a,0}=\beta_{c}+\beta_{0}=0$ and hence, the $1$ in the denominator of (2) arises from $\exp(\beta_{c}+\beta_{0})=1$.

Note that our results on the fixed point computation do not depend on the particular form of the customer choice model. We only require that it is a probability distribution and in the limit as $d_{a,s}\to\infty$ for all $(a,s)\in A\times S$, we have that $\Pi_{a,s}(d)$ tends to zero with a higher than linear rate of convergence. This is important, because otherwise the expected profit-to-go will be unbounded.

For convenience, let the probability that a customer arrives from sub-area $a$ and chooses slot $s$ given prices $d$ be denoted by $p_{a,s}(d):=\lambda\pi(a)\Pi_{a,s}(d)$. We define $p(d):=\left\{p_{a,s}(d)\ \left|\ (a,s)\in A\times S\right.\right\}$ and $P:=\left\{p(d)\ \left|\ d\in D\right.\right\}$. Finally, it is to be understood that sums over $a$ and $s$ are always computed over their entire sets $A$ and $S$, respectively. Similarly, decisions $d$ are made such that $d\in D$.

2.2 Dynamic Programming Formulation

We can express the problem described above as a DP. The expected profit-to-go, $V_{t}(x)$, closely resembles the DP formulation in [6] and we define it as

[TABLE]

i.e. $C(\cdot)$ denotes the terminal condition. The difference $V_{t}(x)-V_{t}(x+1_{a,s})$ represents the value foregone by accepting a additional (discrete spatial) order, which in economic terms is the opportunity cost of an order. Note that – similar to [6] – we ignore any vehicle load capacity constraints in the problem, as they are much less restricting than the time constraints on the delivery slots. Therefore, including the vehicle load capacity constraints would only increase computational costs, but would not substantially improve the decision policy.

For convenience in the sequel, let us define an abstract operator notation which expresses (2.2) in a more compact form:

[TABLE]

3 Concave Continuous Extension Theorem

As the state space $X$ is discrete, it is not possible to establish convexity properties from standard, i.e. continuous, convexity theory. In this section, we therefore first provide some definitions from discrete convex analysis and the assumptions upon which our main results are based. We then state our main result, Theorem 2, and two intermediate results, Theorem 3 and Proposition 4.

3.1 Definitions

Definition 1**.**

We define the set of stochastic vectors in $X$ as

[TABLE]

Definition 2**.**

Let $x\in X$ and let $Q$ be a finite set. Then $Q$ is defined to be an enclosing set of $x$ if $x\in$ conv$(Q)$.

Definition 3**.**

We define $\mathcal{Q}(x)$ as the set of all sets $Q$ enclosing $x$.

Definition 4** (cf. [10, (2.1)]).**

Let $a\in\mathbb{R}^{N}$ and $b\in\mathbb{R}$. Then the concave closure $\tilde{f}:\mathbb{R}^{N}\to\mathbb{R}\cup-\infty$ of a function $f:\mathbb{Z}^{N}\to\mathbb{R}\cup-\infty$ is defined as

[TABLE]

for all $y\in\mathbb{Z}^{N}$ and for all $x\in\mathbb{R}^{N}$.

Definition 5** (cf. [10, Lemma 2.3] and [11, Proposition 2.31]).**

A function $f:\mathbb{Z}^{N}\to\mathbb{R}\cup-\infty$ is concave extensible if and only if any of the following equivalent conditions hold:

• (a)

The evaluations of $f$ coincide with the evaluations of its concave closure $\tilde{f}$, i.e. $f(x)=\tilde{f}(x)$ for all $x\in\mathbb{Z}^{N}$.

• (b)

For all $x\in X$ and for all $Q\in\mathcal{Q}(x)$, the evaluation of $f$ at $x$ does not lie below any possible linear interpolation of $f$ on the points $q\in Q$, i.e. for all $x\in X$, for all $Q\in\mathcal{Q}(x)$ and for all $\mu\in\mathcal{V}_{X}$, such that $x=\sum_{q\in Q}\mu_{q}q$, it holds that

[TABLE]

3.2 Assumptions

We now state the assumptions that our main result builds on.

Assumption 1**.**

The negative cost function $-C$ is concave extensible.

Assumption 2**.**

The marginal cost of an additional, feasible order is always smaller than the maximum marginal profit, i.e. $C(x+1_{a,s})-C(x)\leq\bar{d}+r$, for all $a\in A,s\in S,x\in X$.

Let us define:111Section 4.2.1 details why we have dropped the argument $d$ in $p(d)$ and $p_{a,s}(d)$ and write simply $p$ and $p_{a,s}$ instead.

[TABLE]

for all $x\in X,p\in P$ and $t\in T$.

Assumption 3**.**

For all $t\in T$, we assume that $g_{t}$ is concave extensible in $(x,p)$, for any $V_{t}$ that is concave extensible in $(x,p)$.

Assumption 1 is satisfied for the class of affine functions typically used in the literature [6]. Assumption 2 is not restrictive, as it offers the means to ensure that every additional order can generate profit. Otherwise, the delivery slot prices, which maximise (2.2), would always be $d_{a,s}=\infty$ for all $(a,s)\in A\times S$, resulting in not offering any slots. Assumption 3 intuitively states that $g_{t}(x,p)$, a weighted perturbation of $V_{t}$ in $x$, should be concave extensible. This assumption appears to be strong, but it can always be satisfied by choosing a small enough customer arrival probability $\lambda$. Hence, $p_{a,s}=\lambda\pi(a)\Pi_{a,s}$ and $q_{a,s}^{(p)}=\lambda\pi(a)\Pi_{a,s}^{(q)}$ can be made arbitrarily small and one of the following two cases occurs:

1. Consider that $g_{t}(x,p)$ is strictly concave extensible, by which we mean that the condition for concave extensibility (7) is satisfied with strict inequality, i.e. $V_{t}(x)-\sum_{q\in Q}\mu_{q}V_{t}(q)=\epsilon_{t}(x,Q)>0$. The inequality condition for concave extensibility (7) of $g_{t}(x,p)$ then becomes

[TABLE]

where $(x,p)=\sum_{q\in Q}\mu_{q}\left(q^{(x)},q^{(p)}\right)$. Let us define $w_{t}:=\min\{V_{t}(x+1_{a,s})-V_{t}(x)\}$ as well as $W_{t}:=\max\{V_{t}(q^{(x)}+1_{a,s})-V_{t}(q^{(x)})\}$, where the minimisation is taken with respect to $a$ and $s$ and the maximisation is taken with respect to $q^{(x)}$, $a$ and $s$. Then the inequality in (9) can be tightened to obtain

[TABLE]

As $\epsilon_{t}(x,Q)>0,W_{t}\geq w_{t}$ and $\sum_{a,s}\pi(a)\Pi_{a,s}>0$, there exists a $\lambda>0$ that satisfies the above inequality for all $t\in T$. Therefore, $\lambda$ implicitly depends on $T$, but for simplicity, we will just write $\lambda$.

2. Consider that $\epsilon_{t}(x,Q)=0$, i.e. the points lie on a hyperplane. Then $V_{t}(x+1_{a,s})-V_{t}(x)=V_{t}(q+1_{a,s})-V_{t}(q)$ for all $x\in X,q\in Q,a\in A$ and $s\in S$. Therefore, (9) holds with equality, independently of the particular choice of $\lambda$.

Remark 1**.**

There are a few more special cases, where Assumption 3 is trivially satisfied. For example, if $D$ is a degenerate interval, i.e. it only contains a single $d$, then there is only a single $p$, which also means that $q^{(p)}=p$ for all $q\in Q$ and (9) holds for all $\lambda$.

Similarly, in the case that there is only one delivery sub-area/slot pair $(a,s)$, i.e. $A$ and $S$ are both singleton sets, the inequality condition (9) simplifies to

[TABLE]

As in this scenario $V_{t}$ is a one-dimensional concave extensible function, we can express this inequality in terms of the concave closure of $V_{t}$.

[TABLE]

Noting that $x+p=\sum_{q\in Q}\mu_{q}\left[q^{(x)}+p\right]$, (12) holds, since $\tilde{V}_{t}$ is concave (in the ordinary sense) by Definition 4.

3.3 Statement of Main Results

Based on the aforementioned definitions and assumptions, we formulate our main result:

Theorem 2**.**

Under Assumptions 1, 2 and 3, $V_{t}$ is finite-valued, concave extensible in $x$ for all $t\in T$.

The proof of Theorem 2 mainly depends on the following two results:

Theorem 3**.**

Under Assumption 2, the unique fixed point of (4) is given by

[TABLE]

Proposition 4**.**

Consider Assumption 3 and fix any $t\in T$. If $V_{t}$ is concave extensible in $x$, then $\mathcal{T}V_{t}$ is also concave extensible in $x$.

We now prove Theorem 3, Proposition 4 and Theorem 2 in Section 4.

4 Fixed Point Theorem and Concavity Preservation

4.1 Fixed Point Characterisation

To prove Theorem 3, we first establish a helpful, alternative formulation of (2.2). We then state some supporting lemmata and proceed with the proof. The proofs of all supporting lemmata can be found in the Appendix.

4.1.1 Stochastic Shortest Path Problem Reformulation

In this section, we reformulate (2.2) as an equivalent stochastic shortest path problem, a special version of an undiscounted, finite-state, discrete-time Markov decision problem. We can follow the arguments of [12, Chapter 3] to rewrite (2.2) as

[TABLE]

where $P_{x,y}(d)$ and $\tilde{g}(x,d,y)$ are defined as follows: If $y=x\in X$, then $P_{x,y}(d):=1-\sum_{a,s}p_{a,s}(d)$ and for all $(a,s)\in A\times S$, if $y=x+1_{a,s}$, then $P_{x,y}(d)=p_{a,s}(d)$. In all other cases, $P_{x,y}(d)=0$. Note that the first case is associated with no transition, the second case is a valid, i.e. unit-sized, order and the third group covers all invalid cases.

In a similar way, for all $(a,s)\in A\times S$, if $y=x+1_{a,s}$, then $\tilde{g}(x,d,y)=d_{a,s}+r$ and in all other cases, $\tilde{g}(x,d,y)=0$. For convenience, similarly to [12, Chapter 3], we define $g(x,d):=\sum_{y\in X}P_{x,y}(d)\tilde{g}(x,d,y)$ to simplify (14):

[TABLE]

Let $v^{*}(\cdot)$ be the solution to a different, stationary DP with the same transition probabilities $P_{x,y}(d)$ as in (15), but $g(x,d):=1$ if $x\in X\backslash\{\bar{x}\}$ and $g(x,d):=0$, otherwise.

[TABLE]

Note that $v^{*}(x)\geq 1$ for all $x\in X$. Let us also define $\rho$ as

[TABLE]

Since we have $v^{*}(x)\geq 1$ for all $x\in X$, we conclude that $0\leq\rho<1$. Finally, let

[TABLE]

be a weighted sup-norm of $V_{t}$.

Lemma 5**.**

The mapping defined by the operator $\mathcal{T}$ is contractive with modulus of contraction $\rho$, i.e.

[TABLE]

for all $t\in T$ and $t^{\prime}\in T$.

Hence, we can prove Theorem 3, by showing that there exists a fixed point with the analytic expression in (13).

4.1.2 Proof of Theorem 3

We start with the necessary and sufficient condition for $\mathcal{T}$ to have a fixed point $V^{*}$, which is $V^{*}=\mathcal{T}V^{*}$. This translates into (2.2) as

[TABLE]

Substituting (13) into (20) yields

[TABLE]

The values of all $p_{a,s}(d)$ are non-negative for all $(a,s)\in A\times S$ and for all $d\in D$. The value of $[d_{a,s}-\bar{d}\,]$ is non-positive and [math] only if $d_{a,s}=\bar{d}$ for all $(a,s)\in A\times S$. It follows that the maximum non-negatively weighted sum of the $[d_{a,s}-\bar{d}]$ terms is [math], so (21) holds. As the value function is invariant at $\bar{x}$ and at this point $V^{*}(\bar{x})=V_{\bar{t}+1}(\bar{x})=-C(\bar{x})$, $V^{*}$ is indeed a fixed point of $\mathcal{T}$ for all $x\in X$. ∎

4.2 Preserving Concavity

The structure of this section is similar to the structure of Section 4: We first reformulate (2.2), then we state some supporting lemmata and finally, we prove Proposition 4.

4.2.1 Change of Decision Variables

We start the derivation of Proposition 4 by reformulating (2.2) as a maximisation over $p\in P$ instead of $d\in D$. As shown by [8], this is possible, because the following unique mapping between $p$ and $d$ exists:

[TABLE]

where by solving with respect to $d_{a,s}$ we obtain

[TABLE]

Hence, we can rewrite (2.2) and break it down into two parts:

[TABLE]

where we have defined

[TABLE]

and $g_{t}$ is from (8). This allows us to make the next statement:

Lemma 6**.**

The function $f$ is concave extensible in $(x,p)$.

We are finally ready to prove Proposition 4 showing that $\mathcal{T}$ preserves concave extensibility of $V_{t}$ in $x$.

4.2.2 Proof of Proposition 4 and Theorem 2

By Lemmata 6 and Assumption 3, $f$ and $g$ have continuous extensions $\tilde{f}$ and $\tilde{g}$, which are both jointly concave in $(x,p)$. Therefore, $\tilde{h}(x,p):=\tilde{f}(p)+\tilde{g}(x,p)$ is also jointly concave in $(x,p)$. We define $U(x):=\underset{p}{\max}\,\tilde{h}(x,p)$. This allows us to exploit a standard convex optimisation result (see [11, Proposition 2.22] or [13, Section 3.2.5]), according to which the maximisation with respect to some variables of a continuous multivariate function that is jointly concave in all its variables, yields a concave function. Therefore $U$ is a concave function of $x$.

Repeating the same calculation, now with the discrete $h(x,p):=f(x)+g(x,p)$ in place of $\tilde{h}(x,p)$, i.e. $\mathcal{T}V_{t}(x)=\underset{p}{\max}\,h(x,p)$, note that $h(x,p)=\tilde{h}(x,p)$ for all grid points $x\in X$. Therefore, $\mathcal{T}V_{t}(x)=U(x)$ for all grid points $x\in X$. This shows that $U$ is a continuous extension of $\mathcal{T}V_{t}$, which is concave in $x$. Hence, $\mathcal{T}V_{t}$ is concave extensible in $x$. ∎

Having proved Theorem 3 and Proposition 4 in the previous sections, the stage is set for the proof of Theorem 2.

By the definition of $C$, $V_{\bar{t}+1}$ is finite-valued and by Theorem 3, $V^{*}$ is also finite-valued. Hence for all $x\in X$, the difference $V^{*}(x)-V_{\bar{t}+1}(x)$ is finite-valued. Let us use Lemma 5, according to which for all $t\in T$ and $t^{\prime}\in T$:

[TABLE]

Let $V_{t}=V^{*}=\lim\limits_{t^{\prime}\to-\infty}V_{t^{\prime}}$, where the limit uniquely exists and is well-defined, let $V_{t^{\prime}}=V_{\bar{t}+1}$ and apply $\mathcal{T}$ $N$ times:

[TABLE]

As $\rho<1$, the pointwise difference in $x$ between $V^{*}$ and $V_{t}$ is finite for all $t\in T$, which implies that $V_{t}$ is finite for all $x\in X,t\in T$.

By Assumption 1, $V_{\bar{t}+1}(x):=-C(x)$ for all $x\in X$ is concave extensible. Hence, due to Proposition 4, which is effectively an inductive step, we can conclude that for all $t\in T$, $V_{t}$ is finite-valued, concave extensible in $x$.∎

Due to this result and based on [12, Chapter 3], we can also show that the fixed point (13) is unique. Assume by contradiction that there are two fixed points of (4), $V^{*}$ and $V^{**}$. Substituting $V^{*}$ and $V^{**}$ for $V_{t}$ and $V_{t^{\prime}}$ in (19), respectively we obtain

[TABLE]

Applying $\mathcal{T}$ $N$ times and taking the limit as $N\to\infty$, yields

[TABLE]

By reversing the roles of $V^{*}$ and $V^{**}$, we can conclude that $V^{*}=V^{**}$ and therefore, there can be at most one fixed point.

5 Illustrative Example

We illustrate our findings using a simple numerical example of a 1-area, 2-slot problem. The parameters are listed in Table 1 below.

These parameters yield the terminal condition

[TABLE]

and the fixed point

[TABLE]

for all $x\in X$. To illustrate the contractive mapping property described by (19), we compute $\rho$ numerically by solving the auxiliary DP in (16) and compare it with the running contractive ratio with respect to the fixed point $V^{*}$:

[TABLE]

By choosing $V^{*}$ to be the reference point, we also show that the DP converges to the fixed point as defined in (13). Fig. 1 below illustrates this by showing that $\rho$ is an upper bound for $\rho_{t}$ for all $t\in T$.

We define a measure of discrete concavity

[TABLE]

such that $\mu\in\mathcal{V},\sum_{q\in Q}\mu_{q}V_{t}(q)=V_{t}(x)$ for all $t\in T$. Note that $\epsilon\geq 0$ implies that $V_{t}$ is concave extensible. We compute this quantity by enumeration of all possible enclosing sets and plot the result in Fig. 2, from which it can easily be seen that $\epsilon_{t}(x)\geq 0$ for all $t\in T$.

Finally, we plot the value function $V_{t}$ for $t=\bar{t}-10$ in Fig. 3 below. Note that the value function lies between the terminal condition and the fixed point. When it comes to approximating $V_{t}$, this information can be used to limit the range of basis function parameters, such that the approximated version of $V_{t}$ always lies between the terminal condition and the fixed point.

6 Conclusions and Future Work

We have studied the mathematical properties of the value function of a dynamic program modelling the revenue management problem in attended home delivery exactly. We have shown that the recursive dynamic programming mapping has a unique, finite-valued fixed point and concavity-preserving properties. Hence, we have derived our main result stating that – under certain assumptions – for all time steps in the dynamic program, the value function admits a continuous extension, which is a finite-valued, concave function of its state variables. We have illustrated our findings using a simple numerical example and now conclude with suggestions on how our results can be exploited in the future to obtain closer approximations of the value function.

Recent approaches have estimated $V_{t}$ as an affine function of $x$ for each $t\in T$ [6]. Based on our result, we believe that closer approximations can be found by pursuing different approximation strategies.

One possible direction of future research involves investigating the use of parametric models comprising concave basis functions. This idea can be exploited directly by using the given DP formulation – as suggested in [14, Section 8.2] – or by reformulating the problem as a linear program – as shown by [15]. Note that a priori knowledge of concave extensibility of $V_{t}$ for all $t\in T$ creates some intuitive regularity. Therefore, it can be expected to get good approximations of $V_{t}$ from a relatively small sample size even with simple models.

Another possible direction would be to adapt techniques that fit convex functions (or equivalently concave functions for our purposes) to multidimensional data. For example, [16] and [17] show how data can be fitted by a function defined as the maximum of a finite number of affine functions. More sophisticated examples of convex (concave) function fitting techniques include adaptive partitioning [18] and Bayesian non-parametric regression [19].

Future work will show which strategy will find the best compromise between accuracy of the approximation and computational cost.

Appendix

Proof of Lemma 5.

This proof is based on [12, Chapters 1 and 3]. For any $t\in T$ and $t^{\prime}\in T$ we have:

[TABLE]

∎

Proof of Lemma 6.

By inspection, $f$ is only a function of the continuously-valued variable $p$, so it will be concave extensible in $(x,p)$ if it is concave in $p$ (in the ordinary, continuous sense of concavity). First, note that the function is separable, i.e.

[TABLE]

where we have defined

[TABLE]

Previously, [8] have shown that a structurally similar function is concave in its variables. We adopt their approach – computing the Hessian and showing that it is negative definite – to verify that $f_{a}$ is jointly concave in $\{p_{a,i}\}$ for all $i\in S\cup 0$. We first compute the first-order partial derivatives of $f_{a}$:

[TABLE]

The second-order partial derivatives are:

[TABLE]

Note that the partial derivatives show that $f_{a}$ is independent of all $\{p_{b,s}\}$, for which $a\neq b$. The resulting Hessian $H_{a}$ of $f_{a}$ with its second partial derivatives with respect to $\{p_{a,i}\}$ for all $i\in S\cup 0$ is:

[TABLE]

where we have defined block sub-matrices $A,B,B^{\intercal}$ and $C$ such that $C$ is a scalar corresponding to the last entry of $H_{a}$. Note that $A$ is negative definite, because $p_{a,s}\geq 0$ for all $a\in A,s\in S\cup 0$ and $\beta_{d}^{-1}<0$. We compute the Schur complement of $A$ in $H$:

[TABLE]

As $A$ is negative definite and as $H/A$ is non-positive, $H_{a}$ is negative semi-definite. This implies that for all $a$, $f_{a}$ is concave in $p$. As taking the sum of concave functions preserves concavity (see [13, Section 3.2.1]), $f$ is also concave in $p$. Hence, $f$ is concave extensible in $p$. ∎