On the estimation of discrete choice models to capture irrational customer behaviors

TL;DR

This paper introduces a new estimation method for the Generalized Stochastic Preference model, capturing irrational customer behaviors like halo effects, and demonstrates improved predictive accuracy over traditional models using real-world retail data.

Contribution

It develops an efficient estimation procedure using column generation and a novel dominance rule to model both rational and irrational preferences in customer choice data.

Findings

Boosts predictive accuracy by 12.5% on average.

Effectively models irrational behaviors such as halo effects.

Provides a scalable estimation approach for complex choice models.

Abstract

The Random Utility Maximization model is by far the most adopted framework to estimate consumer choice behavior. However, behavioral economics has provided strong empirical evidence of irrational choice behavior, such as halo effects, that are incompatible with this framework. Models belonging to the Random Utility Maximization family may therefore not accurately capture such irrational behavior. Hence, more general choice models, overcoming such limitations, have been proposed. However, the flexibility of such models comes at the price of increased risk of overfitting. As such, estimating such models remains a challenge. In this work, we propose an estimation method for the recently proposed Generalized Stochastic Preference choice model, which subsumes the family of Random Utility Maximization models and is capable of capturing halo effects. Specifically, we show how to use partially-ranked preferences to efficiently model rational and irrational customer types from transaction data. Our estimation procedure is based on column generation, where relevant customer types are efficiently extracted by expanding a tree-like data structure containing the customer behaviors. Further, we propose a new dominance rule among customer types whose effect is to prioritize low orders of interactions among products. An extensive set of experiments assesses the predictive accuracy of the proposed approach. Our results show that accounting for irrational preferences can boost predictive accuracy by 12.5% on average, when tested on a real-world dataset from a large chain of grocery and drug stores.