Is this normal? A new projection pursuit index to assess a sample against a multivariate null distribution

TL;DR

This paper introduces a new projection pursuit index and visualization method to compare multivariate samples against a reference distribution, aiding in detecting deviations in high-dimensional data.

Contribution

It develops a novel projection pursuit index and visualization technique for high-dimensional sample comparison against a multivariate normal reference.

Findings

Effective visualization of sample deviations in high dimensions

Analytical computation of reference distribution ellipse

Implementation in R package tourr

Abstract

Many data problems contain some reference or normal conditions, upon which to compare newly collected data. This scenario occurs in data collected as part of clinical trials to detect adverse events, or for measuring climate change against historical norms. The data is typically multivariate, and often the normal ranges are specified by a multivariate normal distribution. The work presented in this paper develops methods to compare the new sample against the reference distribution with high-dimensional visualisation. It uses a projection pursuit guided tour to produce a sequence of low-dimensional projections steered towards those where the new sample is most different from the reference. A new projection pursuit index is defined for this purpose. The tour visualisation also includes drawing of the projected ellipse, which is computed analytically, corresponding to the reference distribution. The methods are implemented in the R package, tourr.