Feature Shift Detection: Localizing Which Features Have Shifted via Conditional Distribution Tests

TL;DR

This paper introduces methods to detect and localize specific features responsible for distribution shifts using conditional distribution tests, applicable to multivariate time-series data, with practical results on simulated and real data.

Contribution

It formalizes feature shift localization as multiple conditional distribution hypothesis tests and proposes efficient non-parametric and parametric statistical methods utilizing density model score functions.

Findings

Effective localization of shifted features in multivariate data

Applicable to real-world sensor network scenarios

Compatible with various density models including deep learning models

Abstract

While previous distribution shift detection approaches can identify if a shift has occurred, these approaches cannot localize which specific features have caused a distribution shift -- a critical step in diagnosing or fixing any underlying issue. For example, in military sensor networks, users will want to detect when one or more of the sensors has been compromised, and critically, they will want to know which specific sensors might be compromised. Thus, we first define a formalization of this problem as multiple conditional distribution hypothesis tests and propose both non-parametric and parametric statistical tests. For both efficiency and flexibility, we then propose to use a test statistic based on the density model score function (i.e. gradient with respect to the input) -- which can easily compute test statistics for all dimensions in a single forward and backward pass. Any density model could be used for computing the necessary statistics including deep density models such as normalizing flows or autoregressive models. We additionally develop methods for identifying when and where a shift occurs in multivariate time-series data and show results for multiple scenarios using realistic attack models on both simulated and real world data.