Experimental Assortments for Choice Estimation and Nest Identification

TL;DR

This paper introduces a structured, efficient experiment design for estimating choice models and identifying nests in Nested Logit models, validated through extensive simulations and real-world data from Dream11.

Contribution

It proposes a logarithmic-sample complexity experiment design for choice estimation and a new algorithm for nest identification in Nested Logit models, improving accuracy and interpretability.

Findings

Outperforms heuristic designs in simulations

Successfully identified nests from real user data

Achieved better out-of-sample prediction than feature-based clustering

Abstract

What assortments (subsets of items) should be offered, to collect data for estimating a choice model over $n$ total items? We propose a structured, non-adaptive experiment design requiring only $O(\log n)$ distinct assortments, each offered repeatedly, that consistently outperforms randomized and other heuristic designs across an extensive numerical benchmark that estimates multiple different choice models under a variety of (possibly mis-specified) ground truths. We then focus on Nested Logit choice models, which cluster items into "nests" of close substitutes. Whereas existing Nested Logit estimation procedures assume the nests to be known and fixed, we present a new algorithm to identify nests based on collected data, which when used in conjunction with our experiment design, guarantees correct identification of nests under any Nested Logit ground truth. Our experiment design was deployed to collect data from over 70 million users at Dream11, an Indian fantasy sports platform that offers different types of betting contests, with rich substitution patterns between them. We identify nests based on the collected data, which lead to better out-of-sample choice prediction than ex-ante clustering from contest features. Our identified nests are ex-post justifiable to Dream11 management.