The Four Point Permutation Test for Latent Block Structure in Incidence Matrices

TL;DR

This paper introduces a non-parametric four point permutation test to detect block structures in bipartite graphs representing transactional data, leveraging quasirandom permutation theory and spectral ordering methods.

Contribution

It proposes a novel four point permutation test for block structure detection and discusses efficient computation and enhancement techniques using spectral orders.

Findings

Test effectively measures block structure in bipartite graphs.

Algorithm runs in linear time with respect to edges, scalable with processors.

Spectral ordering can improve block detection accuracy.

Abstract

Transactional data may be represented as a bipartite graph $G:=(L \cup R, E)$, where $L$ denotes agents, $R$ denotes objects visible to many agents, and an edge in $E$ denotes an interaction between an agent and an object. Unsupervised learning seeks to detect block structures in the adjacency matrix $Z$ between $L$ and $R$, thus grouping together sets of agents with similar object interactions. New results on quasirandom permutations suggest a non-parametric \textbf{four point test} to measure the amount of block structure in $G$, with respect to vertex orderings on $L$ and $R$. Take disjoint 4-edge random samples, order these four edges by left endpoint, and count the relative frequencies of the $4!$ possible orderings of the right endpoint. When these orderings are equiprobable, the edge set $E$ corresponds to a quasirandom permutation $\pi$ of $|E|$ symbols. Total variation distance of the relative frequency vector away from the uniform distribution on 24 permutations measures the amount of block structure. Such a test statistic, based on $\lfloor |E|/4 \rfloor$ samples, is computable in $O(|E|/p)$ time on $p$ processors. Possibly block structure may be enhanced by precomputing \textbf{natural orders} on $L$ and $R$, related to the second eigenvector of graph Laplacians. In practice this takes $O(d |E|)$ time, where $d$ is the graph diameter. Five open problems are described.