Covariate-dependent control limits for the detection of abnormal price changes in scanner data

TL;DR

This paper introduces a covariate-dependent control limit method for detecting abnormal price changes in scanner data, incorporating sales volume to improve outlier detection accuracy in consumer price statistics.

Contribution

It proposes a novel outlier detection method that accounts for sales volume, enhancing the detection of abnormal price changes over traditional methods that ignore covariates.

Findings

The new method outperforms existing outlier detection techniques in simulations.

Application to Korean scanner data demonstrates improved detection accuracy.

The variance of log price change is modeled as a smooth function of sales volume.

Abstract

Currently, large-scale sales data for consumer goods, called scanner data, are obtained by scanning the bar codes of individual products at the points of sale of retail outlets. Many national statistical offices use scanner data to build consumer price statistics. In this process, as in other statistical procedures, the detection of abnormal transactions in sales prices is an important step in the analysis. Popular methods for conducting such outlier detection are the quartile method, the Hidiroglou-Berthelot method, the resistant fences method, and the Tukey algorithm. These methods are based solely on information about price changes and not on any of the other covariates (e.g., sales volume or types of retail shops) that are also available from scanner data. In this paper, we propose a new method to detect abnormal price changes that takes into account an additional covariate, namely, sales volume. We assume that the variance of the log of the price change is a smooth function of the sales volume and estimate the function from previously observed data. We numerically show the advantages of the new method over existing methods. We also apply the methods to real scanner data collected at weekly intervals by the Korean Chamber of Commerce and Industry between 2013 and 2014 and compare their performance.