Parameter identification in Markov chain choice models

TL;DR

This paper demonstrates that parameters of a Markov chain choice model can be identified using only small assortments of size two and three, simplifying data collection and estimation processes.

Contribution

It proves that model parameters are identifiable from small assortments, reducing the need for large assortment data in parameter estimation.

Findings

Parameters identifiable from small assortments of size two and three

Efficient algorithm for parameter recovery

Reduces data requirements for model estimation

Abstract

This work studies the parameter identification problem for the Markov chain choice model of Blanchet, Gallego, and Goyal used in assortment planning. In this model, the product selected by a customer is determined by a Markov chain over the products, where the products in the offered assortment are absorbing states. The underlying parameters of the model were previously shown to be identifiable from the choice probabilities for the all-products assortment, together with choice probabilities for assortments of all-but-one products. Obtaining and estimating choice probabilities for such large assortments is not desirable in many settings. The main result of this work is that the parameters may be identified from assortments of sizes two and three, regardless of the total number of products. The result is obtained via a simple and efficient parameter recovery algorithm.

Equations

$${\lambda }_j$$

$$\pi (j,S\mid i)$$

$$\sum _{k\in {\mathcal{N}}_{+}}\pi (j,S\mid k)\cdot {\hat{\rho }}_{i,k}$$

$$\sum _{k\in {\mathcal{N}}_{+}}\pi (j,S\mid k)\cdot {\hat{\lambda }}_k$$

$$\pi (j,S\mid i)$$

$$\pi (j,S)$$

$$=\mathrm{Pr}\left(\text{j is reached before S+∖{j}}\wedge \text{i is not reached before S+}\right)$$

$$+\mathrm{Pr}\left(\text{j is reached before S+∖{j}}\wedge \text{i is reached before S+}\right)$$

$$=\mathrm{Pr}\left(\text{j is reached before (S+∪{i})∖{j}}\right)$$

$$+\mathrm{Pr}\left(\text{j is reached before S+∖{j}}\mid \text{i is reached before S+}\right)$$

$$\cdot \mathrm{Pr}\left(\text{i is reached before S+}\right)$$

$$=\mathrm{Pr}\left(\text{j is reached before (S∪{i})+∖{j}}\right)$$

$$+\mathrm{Pr}\left(\text{j is reached before S+∖{j}}\mid \text{initial state is i}\right)$$

$$\cdot \mathrm{Pr}\left(\text{i is reached before (S∪{i})+∖{i}}\right)$$

$$=\pi (j,S\cup \{i\})+\pi (j,S\mid i)\cdot \pi (i,S).$$

$${\mathbf{h}}_{j,S}$$

$$\dim (V+W)$$

$${\mathbf{h}}_{j,S}^{\top }{\mathbf{e}}_i-{\mathbf{h}}_{j,s}^{\top }{\mathbf{\rho }}_i$$

$$\mathrm{span}\{{\mathbf{h}}_{j,S}\text{mathchar}58j\in S\}\cap \mathrm{span}\{{\mathbf{h}}_{j,S^{\prime }}\text{mathchar}58j\in S^{\prime }\}$$

$$V_{\mathcal{S}}$$

$$V_{S^{\prime }}$$

$$V_{{\mathcal{S}}^{\prime }}$$

$$\dim \left(V_{{\mathcal{S}}^{\prime }}\right)$$

$$V_{\mathcal{S}}\cap V_{S^{\prime }}$$

$$\dim \left(V_{\mathcal{S}}\cap V_{S^{\prime }}\right)$$

$$\dim (W)$$

$$\dim (V_{{\mathcal{S}}^{\prime }})$$

$$=\dim (V_{\mathcal{S}})+\dim (V_{S^{\prime }})-\dim (V_{{\mathcal{S}}^{\prime }}\cap V_{S^{\prime }})$$

$$\ge \dim (V_{\mathcal{S}})+\dim (V_{S^{\prime }})-\max \{1,|S_0\cap S^{\prime }|\}.$$

$$\dim \left(\mathrm{span}\bigcup _{S\in \mathcal{S}}\{{\mathbf{h}}_{j,S}\text{mathchar}58j\in S\}\right)$$

$$\mathcal{S}\text{mathchar}58=\left\{S_{\cap }\cup \{k\}\text{mathchar}58k\in \mathcal{N}\setminus (S_{\cap }\cup \{i\})\right\}.$$

$${\mathbf{h}}_{j,S}$$

$$\dim (\mathrm{span}\{{\mathbf{h}}_{j,S}\text{mathchar}58j\in S_{+}\})$$

$$\mathrm{span}\{{\mathbf{h}}_{j,S}\text{mathchar}58j\in S_{+}\}$$

$$V_{\mathcal{S}}$$

$$V_{S^{\prime }}$$

$$V_{{\mathcal{S}}^{\prime }}$$

$$\dim \left(V_{{\mathcal{S}}^{\prime }}\right)$$

Full text

1 Introduction

In assortment planning, the seller’s goal is to select a subset of products (called an assortment) to offer to a customer so as to maximize the expected revenue. This task can be formulated as an optimization problem given the revenue generated from selling each product, along with a probabilistic model of the customer’s preferences for the products. Such a discrete choice model must capture the customer’s substitution behavior when, for instance, the offered assortment does not contain the customer’s most preferred product.

Our focus in this paper is the Markov chain choice model (MCCM) proposed by Blanchet et al. (2016). In this model, the product selected by the customer is determined by a Markov chain over products where the products in the offered assortment are absorbing states. The current state represents the desired product; if that product is not offered, the customer transitions to another product according to the Markov chain probabilities, and the process continues until the desired product is offered or the customer leaves. MCCM generalizes widely-used discrete choice models such as the multinomial logit model (Luce, 1959; Plackett, 1975), as well as other generalized attraction models (Gallego et al., 2014); it also well-approximates other random utility models found in the literature such as mixed multinomial logit models (McFadden and Train, 2000). At the same time, the MCCM permits computationally efficient unconstrained assortment optimization as well as efficient approximation algorithms in the constrained case (Blanchet et al., 2016; Désir et al., 2015); this stands in contrast to some richer models such as mixed multinomial logit models (Rusmevichientong et al., 2010) and the nested logit model (Davis et al., 2014) for which assortment optimization is generally intractable. This combination of expressiveness and computational tractability makes MCCM very attractive for use in assortment planning.

A crucial step in this overall enterprise—e.g., before assortment optimization may take place—is the estimation of the choice model’s parameters from observational data. Parameter estimation for MCCM is only briefly considered in the original work of Blanchet et al. (2016). In that work, it is shown that the parameters can be determined from the choice probabilities for the all-products assortment, together with the assortments comprised of all-but-one product. This is not satisfactory because it may be unrealistic or unprofitable to offer assortments of such large cardinality. Therefore, it is desirable to be able to determine the parameters from choice probabilities for smaller cardinality assortments. We note that this is indeed possible for simpler choice models such as the multinomial logit model (see, e.g., Train, 2009), but these simpler models are limited in expressiveness—for example, they cannot express heterogeneous substitution behavior.

In this paper, we show that the MCCM parameters can be identified from the choice probabilities for assortments of sizes as small as two and three, independent of the total number of products.111We focus on identifiability because estimation of choice probabilities from observational data is fairly straightforward, especially when the assortments have small cardinality. However, this issue is revisited in Section 5 in the context of sample complexity.

We also give a simple and efficient algorithm for reconstructing the parameters from these choice probabilities.

2 Model and notation

In this section, we describe the Markov chain choice model (MCCM) of Blanchet et al. (2016), along with notations used for choice probabilities and model parameters.

The set of $n$ products in the system is denoted by $\mathcal{N}\mathrel{\mathop{\mathchar 58\relax}}=\{1,2,\dotsc,n\}$. The “no purchase” option is denoted by product [math]. Upon offering an assortment $S\subseteq\mathcal{N}$, the set of possible outcomes is $S_{+}\mathrel{\mathop{\mathchar 58\relax}}=S\cup\{0\}$: either some product in $S$ is purchased, or no product is purchased.

Underlying the MCCM is a Markov chain with state space $\mathcal{N}_{+}$. The (true) parameters of the model are the initial state probabilities $\boldsymbol{\lambda}=(\lambda_{i})_{i\in\mathcal{N}_{+}}$ and the transition probabilities $\boldsymbol{\rho}=(\rho_{i,j})_{(i,j)\in\mathcal{N}_{+}\times\mathcal{N}_{+}}$ (a row stochastic matrix). The transition probabilities satisfy the following properties:

$\rho_{0,0}=1$ and $\rho_{0,j}=0$ for $j\in\mathcal{N}$ (i.e., the “no purchase” state is absorbing). 2. 2.

$\rho_{i,i}=0$ for $i\in\mathcal{N}$ (i.e., no self-loops in product states). 3. 3.

The submatrix $\tilde{}\boldsymbol{\rho}\mathrel{\mathop{\mathchar 58\relax}}=(\rho_{i,j})_{(i,j)\in\mathcal{N}\times\mathcal{N}}$ is irreducible.

We use $\boldsymbol{\rho}_{i}=(\rho_{i,j})_{j\in\mathcal{N}_{+}}$ to denote the $i$-th row of $\boldsymbol{\rho}$.

In MCCM, the customer arrives at a random initial state $X_{1}$ chosen according to $\boldsymbol{\lambda}$. At time $t=1,2,\dotsc$:

• •

If $X_{t}=0$, the customer leaves the system without purchasing a product.

• •

If the product $X_{t}$ is offered (i.e., $X_{t}\in S$), the customer purchases $X_{t}$ and leaves.

• •

If the product $X_{t}$ is not offered (i.e., $X_{t}\notin S$), the customer transitions to a new random state $X_{t+1}$ chosen according to $\boldsymbol{\rho}_{X_{t}}$ and the process continues in time step $t+1$ as if the customer had initially arrived at $X_{t+1}$.

Another way to describe this process is that the Markov chain distribution is temporarily modified so that the states $S_{+}$ are absorbing, and the customer purchases the product upon reaching such a state (or makes no purchase if the state is [math]). The irreducibility of $\tilde{}\boldsymbol{\rho}$ ensures that the customer eventually leaves the system (i.e., an absorbing state is reached). Note that only the identity of the final (absorbing) state is observed, as it corresponds to either a purchase or non-purchase. The $(X_{t})_{t=1,2,\dotsc,}$ themselves do not correspond to observable customer behavior, and hence the model parameters $\boldsymbol{\lambda}$ and $\boldsymbol{\rho}$ cannot be directly estimated.

The choice probabilities are denoted by $\pi(j,S)$ for $S\subseteq\mathcal{N}$ and $j\in S_{+}$: this is the probability that $j$ is the final state in the aforementioned process. Blanchet et al. (2016) relate the choice probabilities and the parameters $\boldsymbol{\lambda}$ and $\boldsymbol{\rho}$ as follows:

[TABLE]

The relations in Equation (1) show that the parameters may be identified from choice probabilities for the assortments $S=\mathcal{N}$ and $S=\mathcal{N}\setminus\{i\}$ for $i\in\mathcal{N}$. These choice probabilities may be directly estimated from observations upon offering such assortments to customers.

3 Main result

The following theorem establishes identifiability of the MCCM parameters from choice probabilities for assortments of sizes as small as two and three.

Theorem 1**.**

There is an efficient algorithm that, for any $r\in\{2,3,\dotsc,n-1\}$, when given as input the choice probabilities $(\pi(j,S))_{j\in S_{+}}$ for all assortments $S\subseteq\mathcal{N}$ of cardinality $r$ and $r+1$ for a Markov chain choice model, returns the parameters $\boldsymbol{\lambda}$ and $\boldsymbol{\rho}$ of the model.

The number of assortments for which the algorithm actually requires choice probabilities is $O(n^{2})$ when $r\leq n/2$, which is far fewer than $\binom{n}{r}+\binom{n}{r+1}$, the total numbers of assortments of sizes $r$ and $r+1$. The details of this bound are shown following the proof of Theorem 1. However, to simplify the presentation, we describe our parameter recovery algorithm as using choice probabilities for all assortments of sizes $r$ and $r+1$.

The main steps of our algorithm, shown as Algorithm 1, involve setting up and then solving systems of linear equations that (as we will prove) determine the unknown parameters $\boldsymbol{\lambda}$ and $(\boldsymbol{\rho}_{i})_{i\in\mathcal{N}}$. (Note that $\boldsymbol{\rho}_{0}$ is already known.) The coefficients of the linear equations are determined by the given choice probabilities via conditional choice probabilities $\pi(j,S\mid i)$ for $S\subseteq\mathcal{N}$ and $i,j\in\mathcal{N}_{+}$, defined as follows:

[TABLE]

Note that the initial state in the MCCM is not observed, so these conditional probabilities cannot be directly estimated. Nevertheless, they can be indirectly estimated via the following relationship between the conditional choice probabilities and the (unconditional) choice probabilities.

Lemma 1**.**

For any $S\subseteq\mathcal{N}$ and $i,j\in S_{+}$,

[TABLE]

Proof.

The cases where $i=j$ ($\Rightarrow\pi(j,S\mid i)=1$) and $i\in S_{+}\setminus\{j\}$ ($\Rightarrow\pi(j,S\mid i)=0$) are clear from the definition in Equation (2). It remains to handle the case where $i\in\mathcal{N}\setminus S$. Fix such a product $i$, and observe that

[TABLE]

The penultimate step uses the Markov property and the case condition that $i\in\mathcal{N}\setminus S$. Rearranging the equation gives the relation claimed by the lemma in this case. ∎

Lemma 1 shows that the conditional choice probabilities for assortments $S$ of size $r$ can be determined from the unconditional choice probabilities of assortments of size $r$ and $r+1$. The systems of linear equations used in Algorithm 1 (Equations (3) and (4)) are defined in terms of these conditional choice probabilities and hence are ultimately defined in terms of the unconditional choice probabilities provided as input to Algorithm 1.

It is clear that the true MCCM parameters $\boldsymbol{\lambda}$ and $\boldsymbol{\rho}$ satisfy the systems of linear equations in Equations (3) and (4). However, what needs to be proved is that they are uniquely determined by these linear equations; this is the main content of the proof of Theorem 1.

4 Proof of Theorem 1

In this section, we give the proof of Theorem 1.

4.1 The case without the “no purchase” option

For sake of clarity, we first give the proof in the case where the “no purchase” option is absent. This can be regarded as the special case where $\lambda_{0}=0$ and $\rho_{i,0}=0$ for all $i\in\mathcal{N}$. So here we just regard $\boldsymbol{\lambda}=(\lambda_{j})_{j\in\mathcal{N}}$ and each $\boldsymbol{\rho}_{i}=(\rho_{i,j})_{j\in\mathcal{N}}$ as probability distributions on $\mathcal{N}$. The general case will easily follow from the same arguments with minor modification.

4.1.1 Proof strategy

We make use of the following result about M-matrices, i.e., the class of matrices $\boldsymbol{A}$ that can be expressed as $\boldsymbol{A}=s\boldsymbol{I}-\boldsymbol{B}$ for some $s>0$ and non-negative matrix $\boldsymbol{B}$ with spectral radius at most $s$. (Here, $\boldsymbol{I}$ denotes the identity matrix of appropriate dimensions.) In particular, the matrix $\boldsymbol{I}-\boldsymbol{\rho}$ is a (singular) M-matrix that is also irreducible.

Lemma 2** **(See, e.g., Theorems 6.2.3 & 6.4.16

in Berman and Plemmons, 1994).

If $\boldsymbol{A}\in\mathbb{R}^{p\times p}$ is an irreducible M-matrix (possibly singular), then every principal submatrix222Recall that a principal submatrix of a $p\times p$ matrix $\boldsymbol{A}$ is a submatrix obtained by removing from $\boldsymbol{A}$ the rows and columns indexed by some set $I\subseteq[p]$. of $\boldsymbol{A}$, other than $\boldsymbol{A}$ itself, is non-singular. If $\boldsymbol{A}$ is also singular, then it has rank $p-1$.

For each $S\in\binom{\mathcal{N}}{r}$ and $j\in S$, define the vector

[TABLE]

For each $i\in\mathcal{N}$, the collection of the vectors $\{\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}S\not\ni i\wedge j\in S\}$ provide the left-hand side coefficients in Equation (3) for $\hat{}\boldsymbol{\rho}_{i}$. We’ll show that the span of these vectors (in fact, a particular subset of them) has dimension at least $n-1$. This is sufficient to conclude that $\boldsymbol{\rho}_{i}$ is the unique solution to the system of equations in Equation (3) because it has at most $n-1$ unknown variables, and it is clear that $\boldsymbol{\rho}_{i}$ satisfies the system of equations. (In fact, there are really only $n-2$ unknown variables, because we can force $\hat{\rho}_{i,i}=0$ and $\hat{\rho}_{i,n}=1-\sum_{k=1}^{n-1}\hat{\rho}_{i,k}$.) For the same reason, it is also sufficient to conclude that $\boldsymbol{\lambda}$ is the unique solution to the system of equations in Equation (4) (where, in fact, we may use all vectors $\{\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}S\in\binom{\mathcal{N}}{r}\wedge j\in S\}$).

4.1.2 Rank of linear equations from a single assortment

We begin by characterizing the space spanned by $\{\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}j\in S\}$ for a fixed $S\in\binom{\mathcal{N}}{r}$. We claim, by Lemma 1, that the vectors in $\{\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}j\in S\}$ are linearly independent. Indeed, if this collection of vectors is arranged in a matrix $[\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}j\in S]$, then the submatrix obtained by selecting rows corresponding to $j\in S$ is the $S\times S$ identity matrix. Thus we have proved

Lemma 3**.**

For any $S\in\binom{\mathcal{N}}{r}$, $\dim\mathinner{\left(\operatorname{span}\{\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}j\in S\}\right)}=|S|=r$.

Note that in the case $r=n-1$, we are done. But when $r<n-1$, the linear equations given by the $\{\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}j\in S\}$ may not uniquely determine the $\boldsymbol{\rho}_{i}$ for $i\in\mathcal{N}\setminus S$. To overcome this, we need to be able to combine linear equations derived from multiple assortments. However, for a sum of subspaces $V$ and $W$,

[TABLE]

unless $V$ and $W$ are orthogonal. In our case, the subspaces $\operatorname{span}\{\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}j\in S\}$ and $\operatorname{span}\{\boldsymbol{h}_{j,S^{\prime}}\mathrel{\mathop{\mathchar 58\relax}}j\in S^{\prime}\}$ for different assortments $S$ and $S^{\prime}$ are not necessarily orthogonal (even if $S$ and $S^{\prime}$ are disjoint). So a different argument is needed.

4.1.3 Rank of linear equations from multiple assortments

Our aim is to show that the intersection of subspaces $V\mathrel{\mathop{\mathchar 58\relax}}=\operatorname{span}\{\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}j\in S\}\cap\operatorname{span}\{\boldsymbol{h}_{j,S^{\prime}}\mathrel{\mathop{\mathchar 58\relax}}j\in S^{\prime}\}$ for different assortments $S$ and $S^{\prime}$ cannot have high dimension. We do this by showing that the intersection is orthogonal to a subspace of high dimension.

For each $i\in\mathcal{N}$, let $\boldsymbol{a}_{i}$ denote the $i$-th row of the matrix $\boldsymbol{A}\mathrel{\mathop{\mathchar 58\relax}}=\boldsymbol{I}-\boldsymbol{\rho}$ (which is an M-matrix). That is, $\boldsymbol{a}_{i}\mathrel{\mathop{\mathchar 58\relax}}=\boldsymbol{e}_{i}-\boldsymbol{\rho}_{i}$, where $\boldsymbol{e}_{i}$ is the $i$-th coordinate basis vector. Recall that if $i\in\mathcal{N}\setminus S$, then $\boldsymbol{\rho}_{i}$ satisfies Equation (3). This fact can be written in our new notation as

[TABLE]

In other words,

Lemma 4**.**

For any $S\in\binom{\mathcal{N}}{r}$, $\operatorname{span}\{\boldsymbol{h}_{j,S}\mathrel{\mathop{\mathchar 58\relax}}j\in S\}\perp\operatorname{span}\{\boldsymbol{a}_{i}\mathrel{\mathop{\mathchar 58\relax}}i\in\mathcal{N}\setminus S\}$.

Now consider two assortments $S$ and $S^{\prime}$, and the intersection of their respective subspaces. It follows from Lemma 4 that

[TABLE]

This orthogonality is the key to lower-bounding the dimension of the sum of these subspaces, which we capture in the following general lemma.

Lemma 5**.**

Let $\mathscr{S}$ be a family of subsets of $\mathcal{N}$, $S^{\prime}$ be a subset of $\mathcal{N}$, and $\mathscr{S}^{\prime}\mathrel{\mathop{\mathchar 58\relax}}=\mathscr{S}\cup\{S^{\prime}\}$. Define the subspaces

[TABLE]

Then

[TABLE]

Proof.

Let $S_{0}\mathrel{\mathop{\mathchar 58\relax}}=\bigcup_{S\in\mathscr{S}}S$. Fix any $\boldsymbol{v}\in V_{\mathscr{S}}\cap V_{S^{\prime}}$. Then, by Lemma 4, $\boldsymbol{a}_{i}^{\scriptscriptstyle{\top}}\boldsymbol{v}=0$ for all $i\in(\mathcal{N}\setminus S_{0})\cup(\mathcal{N}\setminus S^{\prime})=\mathcal{N}\setminus(S_{0}\cap S^{\prime})$. In other words,

[TABLE]

where $W\mathrel{\mathop{\mathchar 58\relax}}=\operatorname{span}\{\boldsymbol{a}_{i}\mathrel{\mathop{\mathchar 58\relax}}i\in\mathcal{N}\setminus(S_{0}\cap S^{\prime})\}$, and

[TABLE]

To determine $\dim(W)$, observe that $W$ is the span of rows of certain rows of the M-matrix $\boldsymbol{A}$. By Lemma 2, the principal submatrix of $\boldsymbol{A}$ corresponding to $\mathcal{N}\setminus(S_{0}\cap S^{\prime})$ is either non-singular (when $S_{0}\cap S^{\prime}\neq\emptyset$) or is $\boldsymbol{A}$ itself; in either case, it has rank $n-\max\{1,|S_{0}\cap S^{\prime}|\}$. Hence,

[TABLE]

as well. Combining the dimension formula with the last two equation displays gives

[TABLE]

The claim now follows from Lemma 3. ∎

4.1.4 Choice of assortments

We now choose a collection of assortments and argue, via Lemma 3 and Lemma 5, that they define linear equations of sufficiently high rank. Specifically, for each $i\in\mathcal{N}$, we need a collection $\mathscr{S}\subset\binom{\mathcal{N}}{r}$ such that each $S\in\mathscr{S}$ does not contain $i$, and

[TABLE]

Lemma 6**.**

Suppose the assortments $S_{1},S_{2},\dotsc,S_{T}\in\binom{\mathcal{N}}{r}$ have a pairwise common intersection $S_{\cap}=S_{t}\cap S_{t^{\prime}}$ for all $t\neq t^{\prime}$, and $|S_{\cap}|=r-1$. Then $\dim\mathinner{(\operatorname{span}\bigcup_{t=1}^{T}\{\boldsymbol{h}_{j,S_{t}}\mathrel{\mathop{\mathchar 58\relax}}j\in S_{t}\})}\geq T+r-1$.

Proof.

Let $d_{\tau}\mathrel{\mathop{\mathchar 58\relax}}=\dim\mathinner{(\operatorname{span}\bigcup_{t=1}^{\tau}\{\boldsymbol{h}_{j,S_{t}}\mathrel{\mathop{\mathchar 58\relax}}j\in S_{t}\})}$ for $\tau\in\{1,2,\dotsc,T\}$. By Lemma 3, we know that $d_{1}=r$. Now, assume $d_{\tau}\geq\tau+r-1$, and use the fact $r\geq 2$ and Lemma 5 to conclude that $d_{\tau+1}\geq d_{\tau}+r-(r-1)=d_{\tau}+1\geq\tau+r$. The claim now follows by induction. ∎

Fix any $i\in\mathcal{N}$ and $S_{\cap}\in\binom{\mathcal{N}\setminus\{i\}}{r-1}$, and observe that $|\mathcal{N}\setminus(S_{\cap}\cup\{i\})|=n-r$. Consider the collection of size-$r$ assortments given by

[TABLE]

These assortments do not contain $i$, they have the common intersection $S_{\cap}$, with $|S_{\cap}|=r-1$, and there are $n-r$ assortments in total. So by Lemma 6, the collection $\mathscr{S}$ satisfies the dimension bound in Equation (6).

As was already argued in the proof strategy, this suffices to establish the uniqueness of the $\boldsymbol{\rho}_{i}$ and $\boldsymbol{\lambda}$ as solutions to the respective systems of linear equations in Equation (3) and Equation (4).

This concludes the proof of Theorem 1 without the “no purchase” option. ∎

4.2 The general case with the “no purchase” option

We now consider the general case, where the “no purchase” option is present. The main difference relative to the previous subsection is that $\boldsymbol{\rho}$ is no longer irreducible, as the “no purchase” state [math] is absorbing. However, the submatrix $\tilde{}\boldsymbol{\rho}=(\rho_{i,j})_{(i,j)\in\mathcal{N}\times\mathcal{N}}$ is irreducible, so $\boldsymbol{I}-\tilde{}\boldsymbol{\rho}$ is an irreducible M-matrix.

The definition of $\boldsymbol{h}_{j,S}$, for $S\subseteq\mathcal{N}$ and $j\in S_{+}$, is now taken to be

[TABLE]

Because the indexing starts at [math], we still define $\boldsymbol{a}_{i}$ to be the $i$-th row of $\boldsymbol{A}=\boldsymbol{I}-\boldsymbol{\rho}$, so $\boldsymbol{a}_{i}=\boldsymbol{e}_{i}-\boldsymbol{\rho}_{i}$. (In particular, $\boldsymbol{a}_{0}$ is the all-zeros vector.)

With these definitions, we have the following analogue of Lemma 3 and Lemma 4:

Lemma 7**.**

For any $S\in\binom{\mathcal{N}}{r}$,

[TABLE]

Here, the key difference is that the dimension is $r+1$, rather than just $r$.

We now establish an analogue of Lemma 5 (which is typographically nearly identical).

Lemma 8**.**

Let $\mathscr{S}$ be a family of subsets of $\mathcal{N}$, $S^{\prime}$ be a subset of $\mathcal{N}$, and $\mathscr{S}^{\prime}\mathrel{\mathop{\mathchar 58\relax}}=\mathscr{S}\cup\{S^{\prime}\}$. Define the subspaces

[TABLE]

Then

[TABLE]

Proof.

The proof is nearly the same as that of Lemma 5. Define $S_{0}\mathrel{\mathop{\mathchar 58\relax}}=\bigcup_{S\in\mathscr{S}}S$ and take $\boldsymbol{v}\in V_{\mathscr{S}}\cap V_{S^{\prime}}$. By Lemma 7, $V_{\mathscr{S}}\cap V_{S^{\prime}}\perp W$, where $W\mathrel{\mathop{\mathchar 58\relax}}=\operatorname{span}\{\boldsymbol{a}_{i}\mathrel{\mathop{\mathchar 58\relax}}i\in\mathcal{N}\setminus(S_{0}\cap S^{\prime})\}$, and

[TABLE]

We now use the fact that $\boldsymbol{I}-\tilde{}\boldsymbol{\rho}$, which is a submatrix of $\boldsymbol{A}$, is an irreducible M-matrix. By Lemma 2, the principal submatrix of $\boldsymbol{A}$ corresponding to $\mathcal{N}\setminus(S_{0}\cap S^{\prime})$ is either non-singular (when $S_{0}\cap S^{\prime}\neq\emptyset$) or is $\boldsymbol{I}-\tilde{}\boldsymbol{\rho}$; in either case, it has rank $n-\max\{1,|S_{0}\cap S^{\prime}|\}$. So we have

[TABLE]

Finishing the proof as in Lemma 5, we have

[TABLE]

where the second inequality uses Lemma 7 (instead of Lemma 3). ∎

The choice of assortments demonstrating the subspace of required dimension is the same as before, except now we show that the dimension is at least $n$. Again, fix some $i\in\mathcal{N}$, and choose the collection of $n-r$ assortments $\mathscr{S}\subseteq\binom{\mathcal{N}}{r}$ as before (described in and directly before Equation (7)). Following the inductive argument in the proof of Lemma 6, but now using Lemma 7 and Lemma 8 (instead of Lemma 3 and Lemma 5), we have

[TABLE]

Since each of the systems of linear equations from Equation (3) and Equation (4) have (at most) $n$ unknown variables, we conclude that the $\boldsymbol{\rho}_{i}$ and $\boldsymbol{\lambda}$ are unique as solutions to their respective systems of linear equations.

This concludes the proof of Theorem 1. ∎

4.3 Total number of assortments required

We now show that the number of assortments for which we need the choice probabilities is $O(n^{2})$ for $r\leq n/2$. Indeed, the construction given above based on Lemma 6 can be used to avoid using all assortments of size $r$ (and $r+1$) in Algorithm 1.

We choose two sets $S_{\cap},S_{\cap}^{\prime}\in\binom{\mathcal{N}}{r-1}$, which shall serve as “common intersection sets” (in the sense used in Section 4.1.4), as follows. The first set $S_{\cap}\in\binom{\mathcal{N}}{r-1}$ is chosen arbitrarily; it serves as the common intersection set for all $i\in\mathcal{N}\setminus S_{\cap}$. The second set $S_{\cap}^{\prime}\in\binom{\mathcal{N}\setminus S_{\cap}}{r-1}$ is chosen arbitrarily as long as it is disjoint from $S_{\cap}$ (which is possible because $r\leq n/2$); it serves as the common intersection set for $i\in S_{\cap}$.

For each $i\in\mathcal{N}\setminus S_{\cap}$, we need the equations for the assortments $S_{\cap}\cup\{k\}$ for all $k\in\mathcal{N}\setminus(S_{\cap}\cup\{i\})$. Obtaining the equations for one such $S_{\cap}\cup\{k\}$ requires choice probabilities for assortments $S_{\cap}\cup\{k\}$ and $S_{\cap}\cup\{k\}\cup\{j\}$ for $j\in\mathcal{N}\setminus(S_{\cap}\cup\{k\})$ as per Lemma 1. In total, for all $i\in\mathcal{N}\setminus S_{\cap}$, we need choice probabilities for $O(n^{2})$ assortments. For the remaining $i\in S_{\cap}$, we use the same argument for the disjoint common intersection set $S_{\cap}^{\prime}$, and thus require the choice probabilities for at most another $O(n^{2})$ assortments.

5 Discussion

Our main result establishes the identifiability of MCCM parameters from choice probabilities for assortments of sizes different from $n-1$ and $n$. This is important because real systems often have cardinality constraints on the assortment sizes. While such constraints are typically considered in the context of assortment optimization (see, e.g., Désir et al., 2015), it is also important in the context of parameter estimation.

One complication of using small size assortments to estimate the MCCM parameters is that the number of different assortments required may be as large as $O(n^{2})$. In contrast, only $n+1$ assortments are needed when the sizes are $n-1$ and $n$. On the other hand, the statistical difficulty of estimating choice probabilities for large assortments may be higher than the same task for smaller assortments. So the possible trade-offs in sample complexity is not straightforward from this analysis. This is an interesting question that we leave to future work.

Acknowledgments

We are grateful to Shipra Agrawal and Vineet Goyal for helpful discussions, and to Vineet for originally suggesting this problem. This work was supported in part by NSF awards DMR-1534910 and IIS-1563785, a Bloomberg Data Science Research Grant, a Sloan Research Fellowship, and the Research Opportunities and Approaches to Data Science (ROADS) grant from the Data Science Institute at Columbia University.