Dynamic Learning with Frequent New Product Launches: A Sequential Multinomial Logit Bandit Problem

TL;DR

This paper introduces a sequential multinomial logit bandit model for online learning in dynamic markets with frequent new product launches, providing algorithms with regret bounds and strategies to mitigate risks in learning new products.

Contribution

It develops a novel SMNL model for tiered recommendations, proposes polynomial-time algorithms for offline and online learning, and extends the model to include learning accuracy constraints for new products.

Findings

Proposed a polynomial-time algorithm for offline preference learning.

Designed an online learning algorithm with quantifiable regret bounds.

Extended the model to ensure learning accuracy for new products.

Abstract

Motivated by the phenomenon that companies introduce new products to keep abreast with customers' rapidly changing tastes, we consider a novel online learning setting where a profit-maximizing seller needs to learn customers' preferences through offering recommendations, which may contain existing products and new products that are launched in the middle of a selling period. We propose a sequential multinomial logit (SMNL) model to characterize customers' behavior when product recommendations are presented in tiers. For the offline version with known customers' preferences, we propose a polynomial-time algorithm and characterize the properties of the optimal tiered product recommendation. For the online problem, we propose a learning algorithm and quantify its regret bound. Moreover, we extend the setting to incorporate a constraint which ensures every new product is learned to a given accuracy. Our results demonstrate the tier structure can be used to mitigate the risks associated with learning new products.

Equations

$$p_i(\mathbb{S})=\{\begin{array}{rlr}& \frac{v_i}{1+{\sum }_{j\in S_1}{v}_j},& \text{ if }i\in S_1\\ & \frac{1}{1+{\sum }_{j\in S_1}{v}_j}\frac{v_i}{1+{\sum }_{j\in S_2}{v}_j},& \text{ if }i\in S_2\\ & \frac{1}{1+{\sum }_{j\in S_1}{v}_j}\frac{1}{1+{\sum }_{j\in S_2}{v}_j},& \text{ if }i=0\\ & 0,& \text{ otherwise,}\end{array}$$

$$\underset{\mathbb{S}}{\max }$$

$$S_k\subseteq X_k,\forall k\in \{1,2\}.$$

$$Re{g}_{\pi }(T;v)=E_{\pi }\left[\sum _{t=1}^T{R}_t({\mathbb{S}}^{\ast },v)-R_t({\mathbb{S}}^t,v)\right],$$

$${\bar{v}}_{i,l}=\frac{1}{T_i(l)}\left(\sum _{\tau \in {\mathcal{T}}_i^1(l)}{\hat{v}}_{i,\tau }^{(1)}+\sum _{\tau \in {\mathcal{T}}_i^2(l)}{\hat{v}}_{i,\tau }^{(2)}\right).$$

$$v_{i,l}^{UCB}:$$

$$+\frac{48\log (K(l-l_{i,0})+1)}{T_i(l)},$$

$$G^{(1)}(\mathbb{S},v):$$

$$G^{(2)}(\mathbb{S},v):=E\left[\sum _{t=1}^{N_2}{R}_t({\mathbb{S}}^{\ast },v)-R_t({\mathbb{S}}^{\prime \prime },v)\right],$$

$$Re{g}_{\pi }(T;v)$$

$$+M\sum _{i\in X}{v}_i(r_{max}-r_i),$$

Full text

 

Keywords*:* sequential, multinomial logit model, bandit, new product, dynamic

1 Introduction

Facing increasingly savvy customers whose preferences are rapidly changing, companies that choose to play it safe by remaining with traditional product lines risk being overtaken by competitors more in tune with their customers. A coping strategy adopted by companies is to frequently launch new products and learn from the market responses. Between the tried-and-true existing products and new products with little or no history, companies face a dilemma - they have to offer new products in order to understand the changing market dynamics so as to improve longer-term profitability, yet they may have to sacrifice short-term profitability. The central question is, how can a company quickly learn customers’ preferences while mitigating the risks inherent in new products?

We approach this question as an online learning task. We consider a seller whose goal is to maximize cumulative profit over a selling horizon $T$. She will introduce several new products at different times during the selling period. For every customer, the seller determines some products to offer111We use “recommend” and “offer” interchangeably in this work. , which may include the existing and/or new products. Based on the customer’s response, the seller updates her belief on the latent customers’ preferences (also known as product valuations), and uses the information to optimize the product selection for the next customer. As we will show in the paper, many new products with relatively low profit will never be offered from a pure profit-maximizing objective.

In reality, companies often intentionally price new products low to gain exposure and to entice customers to give them a try. Thus, many new products may have relatively low profits, yet learning from these product is crucial for understanding customers’ preferences, and enabling companies to make better business decisions in the future. To model such behavior, we impose a constraint, termed “minimum learning criterion”, which requires every new product to be offered and learned to a given accuracy. A direct implication is that the seller will be bearing additional cost of learning as she makes less money from these products. It is natural to ask what can be done to reduce such cost.

We will show that a judicious choice of presenting products is capable of mitigating some costs associated with learning new products. In our setting, products are presented in tiers, indicating the precedence in which customers discover them. For a given customer, a seller first offers the products on the first tier. If none are selected, the seller then presents the second tier, and so on. Priorities are embedded in tiers as product placement affects product visibility. Such product offerings are ubiquitous in the online marketplace. For instance, when companies send multiple emails or app notifications to promote products, or on a website where products are displayed over multiple pages and customers have to take an action (such as clicking on “Next” or ”Load more”) to access the next set of products.

To capture the customer’s responses when recommendations come in tiers, we propose a sequential multinomial logit (SMNL) model, which generalizes the multinomial logit (MNL) model that has been extensively studied in the literature (e.g., Agrawal et al., 2017a, ; Talluri and Van Ryzin,, 2004; Train,, 2009). Besides offering priorities in which products are being shown, we will prove additional benefits of tiered product recommendation, i.e., i) it is capable of achieving higher profit than displaying all products at once, and ii) reducing the profit risks associated with new products.

In this paper, we refer to the online task of learning customers’ preferences through tier-based product recommendations as the SMNL Bandit problem. The contribution of our work is threefold:

We propose a novel SMNL model to capture customer’s sequential choice behavior. For the offline problem with known customers’ preferences, we provide a polynomial-time algorithm to solve the profit-maximization problem, and characterize the properties of the optimal tiered product offering. 2. 2.

For an online setting where new products are frequently launched at different times during the selling horizon, we propose an online learning algorithm for the SMNL Bandit problem, and characterize its regret bound. 3. 3.

We extend the online setting to incorporate a constraint which ensures all new products are learned to a given accuracy, and demonstrate how the tier structure in product presentation can be exploited to mitigate risks with new products.

2 Literature review

The first stream of work that our paper is related to is assortment optimization. It refers to the problem of selecting a set of products to offer to a group of customers so as to maximize the revenue when customers make purchases according to their preferences. It is a central topic in economics, marketing, and the operations management research literature. We refer the reader to Kök et al.,, 2008 for a comprehensive review. Talluri and Van Ryzin,, 2004 is the first paper that models customers’ preferences with the MNL model for the assortment planning problem. Flores et al.,, 2018 study the assortment optimization problem with a different sequential choice model known as the perception-adjusted Luce model and characterize the optimal assortment for the offline problem. Besides the customers’ preferences are modelled differently, we also study the problem in the online setting and investigate the learning policy with new products.

Another related topic is the multi-armed bandit (MAB) problem (e.g., Robbins,, 1985; Sutton et al.,, 1998). Our problem falls under the combinatorial setting (Chen et al.,, 2013) since the retailer’s decision is a combination of different products. A naive approach is to treat each possible combination as an arm. However, the number of arms increases exponentially with the number of products with this approach. Other combinatorial bandit work assuming linear reward (Auer,, 2002; Rusmevichientong and Tsitsiklis,, 2010) or independent rewards (Chen et al.,, 2013) cannot be directly applied to our model. Recent work on assortment optimization (such as Cheung and Simchi-Levi,, 2017; Agrawal et al., 2017a, ; Agrawal et al., 2017b, ; Sauré and Zeevi,, 2013; Rusmevichientong et al.,, 2010) extend the MNL assortment problem from the offline setting to online where customers’ preferences are unknown a priori and need to be learned. Our work is more closely related to Agrawal et al., 2017a, , but with the following key differences. Firstly, we consider multi-tiered assortment. Despite their ubiquity in practice, there is little formal analysis in the literature on either the offline optimization problem or the online learning algorithms. Our work helps to bridge this gap. Secondly, we focus on learning in conjunction with new products launches, where we differentiate two cases depending on whether all new products need to be learned.

3 Problem formulation

In this section, we will formally set up our problem. We will first introduce the SMNL model, which describes the customers’ behavior, and follow by formulating a profit maximization problem that the seller needs to solve.

3.1 Customer’s behavior: SMNL model

Discrete choice models such as the popular MNL model are derived under the assumption that a utility-maximizing customer chooses a product with the highest valuation among a available choice set $\mathbb{S}$ Train, (2009). In a SMNL model, $\mathbb{S}$ consists of multiple tiers of products. For ease of notation, we will present a two-tier model where the choice set consists of two sets, i.e., $\mathbb{S}:=(S_{1},S_{2})$. We will refer to $S_{1}$ and $S_{2}$ as the priority tier and the secondary tier respectively, as products in $S_{1}$ enjoys greater visibility. Note that all our results can be generalized to incorporate more tiers.

Customers arrive at discrete time $t=1,\cdots,T$. For a customer arriving at time $t$, she is presented with a choice set $\mathbb{S}^{t}$ that is selected by the seller. Under the SMNL model, a customer first considers products from the priority tier $S_{1}$. If none are selected, she will then consider the secondary tier $S_{2}$ and decide whether to select any product from $S_{2}$. Note that no-purchase is also one of the choices that the customer can make. The probability that a customer purchases product $i$ is denoted as $p_{i}(\mathbb{S})$ and no-purchase as $p_{0}(\mathbb{S})$, i.e.,

[TABLE]

where $v_{i}$ is the product valuation or customers’ preference for product $i$, which is assumed to be less than 1. For a product from $S_{1}$, its purchase probability follows that of a standard MNL model. On the other hand, the probability of purchasing a product from $S_{2}$, is the joint probability of two events, i.e., the customer has not selected any product from $S_{1}$ and the customer selects a product from $S_{2}$.

3.2 Seller’s profit maximization problem

Knowing customers’ purchase probability as $p_{i}(\mathbb{S})$ when offering $\mathbb{S}$, the seller needs to select a subset of products from all available products to form $S_{1}$ and $S_{2}$. We assume there are two pre-determined sets of product candidates, $X_{1}$ and $X_{2}$. We want to point out that the two candidate sets need not be mutually exclusive, and can completely overlap each other. A seller has the flexibility to assign products as candidates for the priority tier based on sales, trendiness, inventory, and other business criteria.

Denote the profit of product $i$ by $r_{i}$ and the profit obtained from $\mathbb{S}$ by $R(\mathbb{S})$. The expected profit can be expressed as $E[R(\mathbb{S})]=\sum_{i\in S}r_{i}p_{i}(\mathbb{S})=\frac{\sum_{i\in S_{1}}r_{i}v_{i}}{1+\sum_{i\in S_{1}}v_{i}}+\frac{1}{1+\sum_{i\in S_{1}}v_{i}}\frac{\sum_{i\in S_{2}}r_{i}v_{i}}{1+\sum_{i\in S_{2}}v_{i}}.$ The seller’s optimization problem is to select two subsets of products $S_{1}$ and $S_{2}$ from the candidate sets $X_{1}$ and $X_{2}$ respectively. That is,

[TABLE]

We use $\mathbb{S}^{*}=(S_{1}^{*},S_{2}^{*})$ to denote the optimal tiered product offering.

4 Characteristics of the optimal tiered product offering $\mathbb{S}^{*}$

We begin this section with a simple example to compare a two-tiered product offering with its single-tiered counterpart.

**Example 1. ** Suppose there are two products with profit $r_{1}=10,r_{2}=1$ and valuation $v_{1}=0.1,v_{2}=1$ respectively. The optimal one-tier recommendation is to offer both products simultaneously and the corresponding expected profit is given by $E[R(\{1,2\})]=\frac{r_{1}v_{1}+r_{2}v_{2}}{1+v_{1}+v_{2}}=\frac{10*0.1+1*1}{1+0.1+1}=0.952.$ The optimal two-tier recommendation is to offer product 1 on the priority tier and product 2 on the secondary tier. The resulting profit $E[R((\{1\},\{2\}))]=\frac{10*0.1}{1+0.1}+\frac{1}{1.1}\frac{1*1}{1+1}=1.36>E[R(\{1,2\})]$.

This example shows that the tiered structure offers flexibility in presenting products, which translates into higher profit. Intuitively, the tiered recommendation prioritizes products with higher profits to be shown first. We can formalize this observation by analyzing the seller’s problem (3.1) in an offline setting where the product valuation $v_{i}$ is given.

We now introduce two definitions which will help us characterize the properties of the optimal tiered product offering.

Definition 4.1** (Profit-ordered set)**

We call $S_{k}\subseteq X_{k}$ is a profit-ordered set if $\min_{i\in S_{k}}r_{i}\geq\max_{i\in X_{k}\backslash S_{k}}r_{i},$ for $k\in\{1,2\}$.

Definition 4.2** (Profit-ordered by tier)**

*If there exist $i\in S_{1}$ and $j\notin S_{2}$ such that $r_{i}<r_{j}$, then $\mathbb{S}=(S_{1},S_{2})$ is not profit-ordered by tier. Otherwise, it is profit-ordered by tier. *

Example 4.3

Suppose $X_{1}=\{1,2,6,8\}$, $X_{2}=\{3,7,9,10\}$, with profit $r_{i}=i$ for all $i$, then the sets $\mathbb{S}=(\{6,8\},\{7,9,10\})$, $(\{8\},\{3,7,9,10\})$ are both profit-ordered by tier while the sets $\mathbb{S}=(\{2,6,8\},\{7,9,10\})$, $\{(6,8),(9,10)\}$ are not.

Proposition 4.4

The optimal product offering $\mathbb{S}^{*}$ to the optimization problem (3.1) in each tier is a profit-ordered set. In addition, $\mathbb{S}^{*}$ is profit-ordered by tier.

Due to the space constraint, we only include proof sketches for the key results in the paper. All detailed proofs can be found in the supplementary material.

Proof sketch: We show $S_{1}^{*}$ is profit-ordered by contradiction. Supposedly, there exists a $\mathbb{S}^{*}$ where $i\in S_{1}^{*}$ and $r_{i}<E[R(\mathbb{S}^{*})]$, then we show that removing this product will increase the expected profit. Hence, $\mathbb{S}^{*}$ is not optimal. A similar argument is used to show that if $i\notin S_{1}^{*}$, and $r_{i}>E[R(\mathbb{S}^{*})]$, then adding it to the offering will increase the profit. Next, use the same argument to $S_{2}^{*}$ to obtain the desired result.

To prove $\mathbb{S}^{*}$ is profit-ordered by tier, notice that the expected profit of $\mathbb{S}^{*}$ is at least as large as only offering $S_{2}$ since $S=(\emptyset,S_{2})$ is also a feasible solution. Since we have shown that each tier in $\mathbb{S}^{*}$ is a profit-ordered set, i.e., for any $j\in S_{1}^{*}$, $r_{j}\geq E[R(\mathbb{S}^{*})]$, and for any $i\notin S_{2}^{*}$, $r_{i}<E[R(S_{2}^{*})]$. Therefore, $r_{i}\leq E[R(S_{2}^{*})]\leq E[R(\mathbb{S}^{*})]\leq r_{j}$ for any $i\notin S_{2}^{*},j\in S_{1}^{*}$. This completes the proof. $\blacksquare$

Proposition 4.4 implies that a two-tier optimal recommendation can be characterized by a pair of profit thresholds $(\theta_{1},\theta_{2})$ with $\theta_{1}\geq\theta_{2}$, where $r_{i}\geq\theta_{1}$ and $r_{j}\geq\theta_{2}$ for any $i\in S_{1}$ and $j\in S_{2}$. Therefore, the seller’s optimization problem is polynomial-time solvable, as it follows directly from the fact that there are at most $|X_{1}||X_{2}|$ pairs of profit thresholds to enumerate through. In retail, as prices are discrete and often end with 9 or .99, there are far fewer unique price points than the number of products and the actual search space of profit thresholds is significantly smaller.

The profit-ordered structure of the optimal tiered recommendation provides important insights regarding the placement of a new product. We will generalize the result to a setting with multiple tiers.

Proposition 4.5

Denote the optimal recommendation before and after including a new product with profit $r_{m}$ to a candidate set as $\mathbb{S}^{*}=(S^{*}_{1},S^{*}_{2},\cdots,S^{*}_{W})$ and $\hat{\mathbb{S}}^{*}$, respectively. Define $\mathbb{S}^{*}_{j}=(S^{*}_{j},S^{*}_{j+1},\cdots,S^{*}_{W})$. The following properties holds.

1. a.)

$E[R(\hat{\mathbb{S}}_{j}^{*})]\geq E[R(\hat{\mathbb{S}}_{j+1}^{*})]$* for any $j=1,\cdots,W-1$.* 2. b.)

If $E[R(\mathbb{S}^{*}_{j})]<r_{m}<E[R(\mathbb{S}^{*}_{j-1})]$ for some $j$, then $m\in\hat{\mathbb{S}}^{*}$ but $m\notin\hat{S}_{1}^{*}\cup\hat{S}_{2}^{*}\cup\cdots\cup\hat{S}_{j-1}^{*}$. 3. c.)

If $r_{m}<E[R(S^{*}_{W})]$, then $m\notin\hat{\mathbb{S}}^{*}.$

Proposition 4.5 states that, for a two-tier product offering, unless a new product’s profit is higher than $E[R(S_{2}^{*})]$, where $S_{2}^{*}$ refers to what is currently being offered on the secondary tier, it will not be included. Therefore, this product will never be introduced or learned. As discussed in the introduction, many new products could have relatively low profit, but learning is crucial for providing insights to improve long-term profitability. This provides motivation for us to investigate an online learning task with a constraint to ensure all new products are learned to a given accuracy, which we will discuss in Section 6.

5 Learning product valuations

In the previous section, we have assumed that valuations of products are known. In practice, these quantities are not given to the seller and have to be learned.

5.1 Online setup

We consider a general setting where $K$ new products are introduced at different time stamps during a selling horizon $T$. We allow several products to be launched at the same time. We use regret to measure the performance of a learning algorithm, where the regret for a policy $\pi$ is defined as,

[TABLE]

where $\mathbb{S}^{*}$ is the optimal tiered product offering when $\mathbb{v}$ is known, while $\mathbb{S}^{t}$ is the tiered recommendation offered to the customer arriving at time $t$. $R_{t}(\mathbb{S},\mathbb{v})$ denotes the profit accrued at time $t$ when offering recommendation $\mathbb{S}$.

For our learning task, we extend the framework in Agrawal et al., 2017a, which proposed a UCB-based algorithm for an online learning task with a MNL model. We want to emphasize that the tiered structure in the SMNL model significantly complicates the analysis as the decisions across the tiers are interdependent. Next, we will describe a counting process to derive an unbiased estimator of $v_{i}$ for $i\in\mathbb{S}$.

5.2 Unbiased estimator on product valuation

We divide the time horizon into epochs for the priority and the secondary tier respectively, i.e., $\mathcal{L}_{1}$ and $\mathcal{L}_{2}$. Let $\mathcal{L}=\mathcal{L}_{1}\cup\mathcal{L}_{2}$. In each epoch $l\in\mathcal{L}_{k}$ for $k=1,2$, we offer the same product selection $S_{k}^{l}$ for tier $k$ until a no-purchase in $S_{k}^{l}$ occurs. An epoch is labeled as $l$ if and only if $l$ epochs have been completed before $t$. Let $\varepsilon_{l}^{k}$ contain all time steps during epoch $l$ when $S_{k}^{l}$ is shown to a customer.

Example 5.1

Figure 1 illustrates the counting process with an example, which shows the purchase decisions of 9 customers, i.e., $t=1,\cdots,9$. The first customer selects a product from the priority tier, and the second customer selects a product from the secondary tier, and so on. The table in Figure 1 shows how epochs are labeled for different tiers. Here we have $\mathcal{L}_{1}=\{0,1,3,4\}$ and $\mathcal{L}_{2}=\{0,3\}$. For $\mathcal{L}_{1}$, the epoch count at time $t$ is the same as the total number of no-purchases from both tiers before time $t$ . Thus, when $t=6$, epoch $l=3$ since there is a total of 3 no-purchases across both tiers by $t=5$. Note that for the secondary tier $k=2$, we only keep track of the time steps and the epoch count when $S^{l}_{2}$ is shown to a customer (i.e., the customer does not purchase any product from $S^{l}_{1}$). In terms of the time steps for each epoch, we have $\varepsilon_{0}^{1}=\{1,2\}$, $\varepsilon_{1}^{1}=\{3,4,5\}$, $\varepsilon_{3}^{1}=\{6\}$, $\varepsilon_{4}^{1}=\{7,8,9\}$, $\varepsilon_{0}^{2}=\{2,5\}$, and $\varepsilon_{3}^{2}=\{6,9\}$.

For any time step $t$, we use $c_{t}^{k}$ to denote the purchase decision of customer $t$ on tier $k$, i.e., $1(c_{t}^{k}=i)=1$ if the consumer purchased product $i\in S_{k}$, and 0 for a no-purchase. For any product $i\in S_{1}^{l}$ and $j\in S_{2}^{l}$, define $\hat{v}_{i,l}^{(1)}=\sum_{t\in\varepsilon_{l}^{1}}1(c_{t}^{1}=i)$ and $\hat{v}_{j,l}^{(2)}=\sum_{t\in\varepsilon_{l}^{2}}1(c_{t}^{2}=j)$ as the number of times a product $i$ is purchased in epoch $l$ as part of the primary or secondary tier selections respectively.

Let $\mathcal{T}_{i}^{k}(l)$ be the set of epochs which contain product $i$ in tier $k$ offering before epoch $l$. Define $T_{i}^{k}(l)=|\mathcal{T}_{i}^{k}(l)|$, which denotes the number of epochs which contain $i$ in tier $k$ offering before epoch $l$. Let $T_{i}(l)=T_{i}^{1}(l)+T_{i}^{2}(l)$, as the total number of epochs which contain $i$ in the tiered recommendation before epoch $l$. We compute $\bar{v}_{i,l}$ as the average number of times product $i$ is purchased per epoch, i.e.,

[TABLE]

Lemma 5.2

$\hat{v}_{i,l}^{(k)}$* are i.i.d. geometric random variables with parameter $\frac{1}{1+v_{i}}$ for any $l$ and $k=1,2$. Therefore, they are unbiased i.i.d. estimators of $v_{i}$.*

5.3 Learning algorithm for SMNL bandit

Define the upper confidence bound on $v_{i}$ as the follows,

[TABLE]

where $l_{i,0}$ is the initial launch epoch of product $i$, $\bar{v}_{i,l}$ is defined in Equation (5.1), and $K$ is the total number of products.

We briefly describe our UCB-based algorithm: In each epoch $l$, we use $\mathbb{v}_{l}^{UCB}$ to compute the optimal product offering . Denote $\tilde{\mathbb{S}}^{l}$ as the optimal product set when the value of products is $\mathbb{v}^{UCB}_{l}$ and $\mathbb{S}^{*}$ is the optimal set selected from the entire candidate sets including the new product. To bound the profit difference between $\mathbb{S}^{*}$ and $\tilde{\mathbb{S}}^{l}$, we derive the following result.

Lemma 5.3

Assume $0\leq v_{i}\leq v_{i}^{UCB}$ for all $i=1,\cdots,K$. Suppose $\mathbb{S}^{*}$ is an optimal tiered recommendation when the parameters of SMNL model are given by $\mathbb{v}$. Then $E[R(\mathbb{S}^{*},\mathbb{v}^{UCB})]\geq E[R(\mathbb{S}^{*},\mathbb{v})]$.

Lemma 5.3 is a key step in the regret analysis for this UCB-based algorithm. With Lemma 5.3, on the “large probability” event that $0\leq v_{i}\leq v_{i}^{UCB}$ for all $i=1,\cdots,K$, we can bound the difference $E[R(\mathbb{S}^{*},\mathbb{v})]-E[R(\tilde{\mathbb{S}}^{l},\mathbb{v})]$ by $E[R(\tilde{\mathbb{S}}^{l},\mathbb{v}^{UCB})]-E[R(\tilde{\mathbb{S}}^{l},\mathbb{v})]$. We will expand the regret analysis with more details in next section, where we impose an additional constraint to our learning task, as the current setting is a special case when the constraint is absent.

6 Regret analysis with the minimum learning criterion

As we have discussed in Section 4, by default a new product will only be included in the product offering if its profit $r_{m}\geq E[R(S_{2})]$, where $S_{2}$ is the current product offering at the secondary tier. In other words, new products with profit $r_{m}<E[R(S_{2})]$ will never be offered and and deprived of the learning opportunity. To have a more realistic setting, we will formally define a minimum learning constraint. We will then investigate a learning algorithm and quantify its resulting regret, starting with a single new product and later generalize to multiples.

6.1 Minimum learning criterion

We impose a constraint in our learning task to ensure that every product will be offered for at least a number of times to allow us to learn its valuation to a certain accuracy. More specifically, we require the estimated valuation $\bar{v}_{i}$ of every new product to be within $\epsilon$ to the true $v_{i}$ with a probability which is at least $1-\alpha$, where $\epsilon$ and $\alpha$ are two pre-determined parameters. We derive the following lemma which specifies the number of epochs $M$ needed to achieve a given level of estimation accuracy.

Lemma 6.1** (Minimum learning criterion)**

For any $\epsilon$ and $\alpha>0$, if the number of epochs $M\geq\frac{192\log(2/\alpha+1)}{(-1+\sqrt{1+4\epsilon})^{2}}$, then $\bar{v}_{i}$ is within the $\epsilon$ confidence bound of $v_{i}$ with probability at least $1-\alpha$. That is, $P(|\bar{v}_{i,l}-v_{i}|>\epsilon)<1-\alpha$ if $T_{i}(l)>\frac{192\log(2/\alpha+1)}{(-1+\sqrt{1+4\epsilon})^{2}}$.

We want to emphasize that the constraint only affects a subset of new products which are otherwise excluded from being offered due to their relatively low profitability. Once they are offered and $M$ samples have been collected, they will be dropped out from future product recommendations. On the other hand, new products (along with some existing products) with relatively high profit will continuously be offered after $M$ epochs and the estimation on their product valuations will be further improved. This is echoing what typically happens after product launches, where companies choose to continue or stop certain new products based on market response.

6.2 Learning with $r_{m}<E[R(S_{2})]$

In this section, we focus on with a setting when a single new product with low profit is launched in the middle of a selling horizon. Part of our goal is to determine the best way to include this product into learning.

By Proposition 4.5, this low-profit product will be excluded from learning by default. In order to satisfy the minimum learning criterion, this new product will have to be offered for $M$ epochs, where $M$ is determined by Lemma 6.1. There are two possible strategies for us to learn this new product, i.e., either assigning it to the priority tier or the secondary tier.

The answer to which is a better strategy is not immediately clear: While the duration of an epoch is shorter when a product is placed on the priority tier, it could also mean that more of this product will be purchased. Hence, more profit loss and higher regret. On the other hand, even though a product placed on the secondary tier might make fewer sales, the duration of a single epoch could be much longer and the resulting regret could still be high since other products (in addition to the new product) also contribute to the total regret. We now formally compare the two strategies by quantifying the corresponding regrets incurred during a single epoch.

Strategy 1: Assigning new product to the priority tier

Let $S_{1}^{\prime}=S_{1}\cup\{m\}$, $S_{2}^{\prime}=S_{2}$, and $\mathbb{S}^{\prime}=(S_{1}^{\prime},S_{2}^{\prime})$. Let $N_{1}$ denote the number of times $S^{\prime}_{1}$ has been shown to customers until a no-purchase occurs. Note that $N_{1}$ follows the geometric distribution with mean $1+\sum_{j\in S_{1}^{\prime}}v_{j}$, which depends on the valuation of all products in $S_{1}^{\prime}$.

Define the regret function during one epoch when the new product is included in the first tier as $G^{(1)}(\mathbb{S},\mathbb{v})$, i.e.,

[TABLE]

where $\mathbb{S}=(S_{1},S_{2})$ and $\mathbb{S}^{\prime}=(S_{1}^{\prime},S_{2})=(S_{1}\cup\{m\},S_{2})$.

Strategy 2: Assigning new product to the secondary tier

Let $S_{1}^{\prime\prime}=S_{1}$, $S_{2}^{\prime\prime}=S_{2}\cup\{m\}$, and $\mathbb{S}^{\prime\prime}=(S_{1}^{\prime\prime},S_{2}^{\prime\prime})$. $N_{2}$ denotes the number of times $\mathbb{S}^{\prime\prime}$ has been shown to customers until a no-purchase from the entire product offering (i.e., both tiers). $N_{2}$ follows the geometric distribution with mean $(1+\sum_{j\in S_{1}}v_{j})(1+\sum_{j\in S_{2}^{\prime}}v_{j})$.

Similarly, we define the corresponding regret function as follows,

[TABLE]

where $\mathbb{S}=(S_{1},S_{2})$ and $\mathbb{S}^{\prime\prime}=(S_{1},S_{2}^{\prime})=(S_{1},S_{2}\cup\{m\})$.

To compare the two strategies, we first need to determine the optimal action under a given strategy, then evaluate its “best” loss. The strategy which yields the lower regret is then considered a “better” strategy. Let $\mathbb{Q}^{*}$ and $\mathbb{Q}^{\prime*}$ denote the optimal solution that minimizes the regret $G^{(1)}$ and $G^{(2)}$, respectively, i.e., $\mathbb{Q}^{*}=\operatorname*{\rm argmin}_{\mathbb{S}}G^{(1)}(\mathbb{S},\mathbb{v})$ and $\mathbb{Q^{\prime}}^{*}=\operatorname*{\rm argmin}_{\mathbb{S}}G^{(2)}(\mathbb{S},\mathbb{v})$.

Theorem 6.2

*The optimal solution to $G^{(1)}$ and $G^{(2)}$ is the same as $\mathbb{S}^{*}$. That is, $\mathbb{Q}^{*}=\mathbb{Q}^{\prime*}=\mathbb{S}^{*}.$ In addition, we have $G^{(1)}(\mathbb{S}^{*},\mathbb{v})\geq G^{(2)}(\mathbb{S}^{*},\mathbb{v})=v_{m}(E[R(S_{2}^{*})]-r_{m}).$ *

The implication of Theorem 6.2 is twofold. Firstly, it shows that the optimal offerings excluding the new product are identical for both strategies, irrespective of which tier the new product has been added to. In addition, they are also the same as the optimal offering $\mathbb{S}^{*}$ before the new product is added. In other words, there is no need to resolve the optimization problem with the added new product. Thus, it provides a simple learning algorithm for a new product with $r_{m}<E[R(S_{2}^{*})]$: It is optimal to just add it to the secondary tier of the existing optimal product offering to satisfy the learning criterion.

Secondly, Theorem 6.2 also shows that with this optimal product offering $\mathbb{S}^{*}$, the regret is lower when the new product is added to the secondary tier. This result highlights the advantage of showcasing product recommendations in multiple tiers, in the sense we incur a smaller loss by displaying new products with higher risks (i.e., lower profit) on tiers with lower priorities.

6.3 Learning with multiple new products

This section focuses on a general setting similar to the one addressed in Section 5.1, except with the minimum learning constraint in place.

We propose Algorithm 1 to dynamically offer the recommendation which simultaneously explores and exploits. In Algorithm 1, for each epoch $l$, we compute the optimal tiered recommendation $\tilde{\mathbb{S}}^{l}$ given valuation $\mathbb{v}_{l}^{UCB}$. Based on Proposition 6.2, for any new product $i\in\mathcal{N}$ which is not included in $\tilde{\mathbb{S}}^{l}$, we add it to the second tier $\tilde{S}^{l}_{2}$. At the end of each epoch, we update $\bar{\mathbb{v}}_{l}$ and $\mathbb{v}^{UCB}_{l}$, which will be used to compute the recommendation for the next epoch.

We are now ready to present an upper bound on the regret for Algorithm 1. We provide a proof sketch here and the detailed proof can be found in the Supplementary Material.

Theorem 6.3** (Performance bound for Algorithm 1)**

The regret during time $[0,T]$ is bounded above by

[TABLE]

for some constant $C$, where $r_{max}$ is the highest profit of products among $X$, and $K$ is the total number of products.

Proof sketch: We first rewrite the regret in terms of the epochs. Note that one learning epoch on the secondary tier may correspond to multiple learning epochs on the priority tier. Let $\kappa(l)$ denote as a set of epochs on tier 1 which corresponds to epoch $l\in\mathcal{L}_{2}$. In Example 2 as shown in Figure 1, we have $\kappa(0)=\{0,1\}$, $\kappa(3)=\{3,4\}$. Thus, the regret until time $T$ can be expressed $Reg_{\pi}(T;\mathbb{v})=E_{\pi}[\sum_{l\in\mathcal{L}_{2}}\sum_{j\in\kappa(l)}\sum_{t\in\varepsilon_{j}^{1}}(R_{t}(\mathbb{S}_{j}^{*},\mathbb{v})-R_{t}((\tilde{S}^{j}_{1},\tilde{S}^{l}_{2}\cup H_{l}),\mathbb{v}))]$, where the set $H_{l}$ denotes the set of new products with low profit which are added to the second tier at epoch $l\in\mathcal{L}_{2}$ to satisfy the minimum learning criterion.

Define the “large probability” event $A_{l}=\bigcap_{i=1}^{K}\{v_{i,l}^{UCB}-C_{1}\sqrt{\frac{v_{i}\log(K(l-l_{i,0})+1)}{T_{i}(l)}}-C_{2}\frac{\log(K(l-l_{i,0})+1)}{T_{i}(l)}<v_{i}<v_{i,l}^{UCB}\}.$ Meanwhile, by Lemma 5.3, we have $E[R_{t}(\tilde{\mathbb{S}}^{l},\mathbb{v})]\leq E[R_{t}(\mathbb{S}_{l}^{*},\mathbb{v})]\leq E[R_{t}(\mathbb{S}_{l}^{*},\mathbb{v}^{UCB})]\leq E[R_{t}(\tilde{\mathbb{S}}^{l},\mathbb{v}^{UCB})]$. Thus, conditional on the event $A_{l}$ and Lemma 5.3, we can show that $E[R_{t}(\mathbb{S}_{l}^{*},\mathbb{v})-R_{t}((\tilde{S}_{1}^{l},\tilde{S}_{2}^{l}\cup H_{l}),\mathbb{v})]$ can be bounded above by $E[R_{t}(\tilde{\mathbb{S}}^{l},\mathbb{v}_{l}^{UCB})-R_{t}(\tilde{\mathbb{S}}^{l},\mathbb{v})]+E[R_{t}(\tilde{\mathbb{S}}^{l},\mathbb{v})-R_{t}((\tilde{S}_{1}^{l},\tilde{S}_{2}^{l}\cup H_{l}),\mathbb{v})]$.

We see that the regret consists of two parts: The first term can be bounded above by

$E[\sum_{l\in\mathcal{L}_{1}}\sum_{i\in\tilde{S}_{1}^{l}}r_{i}(v_{i,l}^{UCB}-v_{i})+\sum_{l\in\mathcal{L}_{2}}\sum_{i\in\tilde{S}_{2}^{l}}r_{i}(v_{i,l}^{UCB}-v_{i})].$ The second term can also be bounded since each product will be included in the set $H$ for at most $M$ times.

Combined this result on the “large probability” event $A_{l}$ with the error on the measure of “small probability” event $A_{l}^{c}$, the upper bound of the regret can be obtained. $\blacksquare$

We want to point out Algorithm 1 can be easily extended to include more than two tiers, and Theorem 6.3 will continue to hold. The regret bound in Theorem 6.3 consists of three terms, where the first two terms account for the estimation error on product valuation, while the third term is linear with $M$, representing the price one has to pay in order to include new products with low profit into learning. When $M=0$, Theorem 6.3 provides the regret bound for the case without the minimum learning criterion, which is a special case discussed in Section 5.3.

7 Numerical experiments

In this section, we conduct three experiments. We first investigate the robustness of Algorithm 1. Next, we compare Algorithm 1 that simultaneously explores and exploits with a benchmark algorithm which separates the two phases. Lastly, we compare our algorithm with an alternative strategy for learning new products.

Experiment 1 (Robustness study)

We consider a setting where $X$ contains 80 products with profit $r_{i}$ uniformly distributed on [0,1] and 20 products with $r_{i}$ uniformly distributed [0,0.2]. We compare four scenarios, when the product valuation $v_{i}$ is uniformly distributed on [0,0,1], [0,0.2], [0,0.3], and [0,0.5]. A new product is introduced after every 800 time steps. We set $M=100$ for the minimum learning criteria.

Figure 2 shows the results based on 10 independent simulations for different distributions of $\mathbb{v}$. The average regrets are 129.87, 243.38, 348.31, and 620.14 for the four scenarios. Notice that both the mean and variance of the regret are increasing with the support of $\mathbb{v}$. It implies that the learning process is harder when the product valuations $\mathbb{v}$ lie on a larger support and have higher variability.

Experiment 2 (Comparison with a explore-then-exploit benchmark)

The benchmark we consider is adapted from Sauré and Zeevi,, 2013. As shown in Section 4, there are at most $|X_{1}||X_{2}|$ candidates which are profit-ordered by tier. In the exploration phase of the benchmark algorithm, every candidate whose profit is higher than the current optimum is offered for at least $\gamma\log(t)$ times, where $\gamma$ is a tuning parameter. In the exploitation phase, the algorithm uses the estimated parameters to determine a tiered offering with the highest expected profit and offer it to all customers.

For the experiment, consider the setting that $X$ contains 12 products, where the profit $r_{i}$ of 8 of them are uniformly distributed on [0,1], and that of 4 products on [0,0.2]. The valuation $v_{i}$ is uniformly distributed on [0,0.1]. For ease of comparison, all products are launched at $t=0$. Set $M=100$.

Figure 3 shows the results based on 10 independent simulation. It depicts the superiority of our algorithm over the benchmark, where the average regrets are 14.39 and 247.78 under Algorithm 1 and the benchmark respectively.

Experiment 3 (Comparison with an alternative learning strategy for new products)

We have shown in Algorithm 1 that new products with profit lower than $E[R(S_{2}^{*})]$ will be added to the secondary tier. In this experiment, we compare it with an alternative strategy where those new products with low profit will be randomly added to either tier with equal probability for learning. To be precise, we consider a setting where $X_{1}$ contains 20 products with profit uniformly distributed on [0.5,1] and valuation on [0,0.1]. $X_{2}$ contains 30 products with profit uniformly distributed on [0,0.6] and valuation on [0,0.2]. We compute the optimal product offering as the current offering based on these values. Next, we assume 15 new products with profit uniformly distributed on [0,0.55] and valuation on [0,0.3] are launched at time $t=0$. For the benchmark, new products with profit below $E[R(S_{2}^{*})]$ will be randomly added to one of the tiers. Set $M=300$.

As shown in Figure 4, the average regrets are 102.21 under Algorithm 1 and 178.00 under the alternative strategy. It highlights the benefit of having a tiered offering as one could use the secondary tier to mitigate some profit risk when learning with new products.

8 Conclusion

In this work, we studied a product selection problem with a SMNL model which specifies the order in which products are being presented. For the offline setting where the product valuations are known, a polynomial-time solvable algorithm was provided. For the online setting, we analyzed a novel setup where multiple new products could arrive in the middle of a selling period. Depending on the presence of the minimum learning criterion, we proposed an online algorithm and characterized its regret.

There are several future directions of this work. For instance, products’ valuations may vary with time, especially for fashion and technology products. Thus, there is a need for an online algorithm that learns the dynamic valuations. In addition, it would be interesting to utilize customer attribute data and historical sales data to provide personalized recommendations.