The Top-K Tau-Path Screen for Monotone Association

TL;DR

This paper introduces the top-K tau-path (TKTP) method, which efficiently identifies subpopulations with monotone associations in large datasets, improving upon previous methods by enabling analysis of thousands of variable pairs.

Contribution

The paper presents a novel TKTP algorithm that enhances the detection of subpopulations with monotone associations, scalable to large datasets, with detailed computational analysis and practical demonstrations.

Findings

Algorithm efficiently identifies subpopulations with high association.

Scalability to thousands of variable pairs and large sample sizes.

Simulation studies confirm accuracy across various settings.

Abstract

A pair of variables that tend to rise and fall either together or in opposition are said to be monotonically associated. For certain phenomena, this tendency is causally restricted to a subpopulation, as, for example, an allergic reaction to an irritant. Previously, Yu et al. (2011) devised a method of rearranging observations to test paired data to see if such an association might be present in a subpopulation. However, the computational intensity of the method limited its application to relatively small samples of data, and the test itself only judges if association is present in some subpopulation; it does not clearly identify the subsample that came from this subpopulation, especially when the whole sample tests positive. The present paper adds a "top-K" feature (Sampath and Verducci (2013)) based on a multistage ranking model, that identifies a concise subsample that is likely to contain a high proportion of observations from the subpopulation in which the association is supported. Computational improvements incorporated into this top-K tau-path (TKTP) algorithm now allow the method to be extended to thousands of pairs of variables measured on sample sizes in the thousands. A description of the new algorithm along with measures of computational complexity and practical efficiency help to gauge its potential use in different settings. Simulation studies catalog its accuracy in various settings, and an example from finance illustrates its step-by-step use.