a-TMFG: Scalable Triangulated Maximally Filtered Graphs via Approximate Nearest Neighbors

TL;DR

The paper introduces a scalable approximation algorithm for constructing triangulated maximally filtered graphs using k-nearest neighbors, enabling analysis of large datasets where traditional methods are infeasible.

Contribution

It proposes the a-TMFG algorithm that reduces memory and runtime constraints by leveraging approximate nearest neighbors and on-the-fly correlation estimation.

Findings

Effective on datasets with millions of observations

Robust to parameter variations and noise

Enables graph construction for large-scale data analysis

Abstract

The traditional Triangular Maximally Filtered Graph (TMFG) construction requires pre-computation and storage of a dense correlation matrix; this limits its applicability to small and medium-sized datasets. Here we identify key memory and runtime complexity challenges when using TMFG at scale. We then present the Approximate Triangular Maximally Filtered Graph (a-TMFG) algorithm. This is a novel approach to scaling the construction of artificial graphs from data inspired by TMFG. The method employs k-Nearest Neighbors Graphs (kNNG) for initial construction, and implements a memory management strategy to search and estimate missing correlations on-the-fly. This provides representations to control combinatorial explosion. The algorithm is tested for robustness to the parameters and noise, and is evaluated on datasets with millions of observations. This new method provides a parsimonious way to construct graphs for use-cases where graphs are used as input to supervised and unsupervised learning but where no natural graph exists.