Dirichlet Process Mixtures of Order Statistics with Applications to Retail Analytics

TL;DR

This paper introduces a nonparametric Bayesian approach using Dirichlet Process Mixtures with an Exponentiated Weibull kernel to model order statistic vectors, specifically applied to retail data with missing information on product competition.

Contribution

It develops a novel Dirichlet Process Mixture Model for order statistics, enabling flexible modeling and clustering of high-dimensional, sparse, and lossy data in retail analytics.

Findings

Identifies distinct product behavior clusters

Demonstrates flexible modeling of order statistics

Reveals market competition patterns

Abstract

The rise of "big data" has led to the frequent need to process and store datasets containing large numbers of high dimensional observations. Due to storage restrictions, these observations might be recorded in a lossy-but-sparse manner, with information collapsed onto a few entries which are considered important. This results in informative missingness in the observed data. Our motivating application comes from retail analytics, where the behaviour of product sales is summarised by the price elasticity of each product with respect to a small number of its top competitors. The resulting data are vectors of order statistics, due to only the top few entries being observed. Interest lies in characterising the behaviour of a product's competitors, and clustering products based on how their competition is spread across the market. We develop nonparametric Bayesian methodology for modelling vectors of order statistics that utilises a Dirichlet Process Mixture Model with an Exponentiated Weibull kernel. Our approach allows us added flexibility for the distribution of each vector, while providing parameters that characterise the decay of the leading entries. We implement our methods on a retail analytics dataset of the cross-elasticity coefficients, and our analysis reveals distinct types of behaviour across the different products of interest.