Measuring the Stability of Learned Features

TL;DR

This paper develops methods to assess the stability of features learned from non-rectangular data using statistical learning, introducing visualization tools and demonstrating their application on spatial proteomics data.

Contribution

It proposes new strategies for evaluating feature stability in complex datasets, extending existing techniques to non-rectangular data structures.

Findings

Stability curves effectively visualize feature reliability.

Simulations demonstrate the power of proposed methods.

Application to proteomics data shows practical utility.

Abstract

Many modern datasets don't fit neatly into $n \times p$ matrices, but most techniques for measuring statistical stability expect rectangular data. We study methods for stability assessment on non-rectangular data, using statistical learning algorithms to extract rectangular latent features. We design controlled simulations to characterize the power and practicality of competing approaches. This motivates new strategies for visualizing feature stability. Our stability curves supplement the direct analysis, providing information about the reliability of inferences based on learned features. Finally, we illustrate our approach using a spatial proteomics dataset, where machine learning tools can augment the scientist's workflow, but where guarantees of statistical reproducibility are still central. Our raw data, packaged code, and experimental outputs are publicly available.