Communication Complexity of Distributed High Dimensional Correlation Testing

TL;DR

This paper investigates the communication complexity of testing correlation in high-dimensional distributed data, establishing tight bounds and showing the efficiency of direct testing over estimation methods.

Contribution

It provides tight bounds on the communication needed for distributed correlation testing and demonstrates the suboptimality of estimate-and-test approaches.

Findings

Order $d/\tau^2$ bits are sufficient and necessary for correlation testing.

Estimate-and-test approach requires roughly $d^2/\tau^2$ bits, which is suboptimal.

Bounds are tight for the one-dimensional one-way communication case.

Abstract

Two parties observe independent copies of a $d$-dimensional vector and a scalar. They seek to test if their data is correlated or not, namely they seek to test if the norm $\|\rho\|_2$ of the correlation vector $\rho$ between their observations exceeds $\tau$ or is it $0$. To that end, they communicate interactively and declare the output of the test. We show that roughly order $d/\tau^2$ bits of communication are sufficient and necessary for resolving the distributed correlation testing problem above. Furthermore, we establish a lower bound of roughly $d^2/\tau^2$ bits for communication needed for distributed correlation estimation, rendering the estimate-and-test approach suboptimal in communication required for distributed correlation testing. For the one-dimensional case with one-way communication, our bounds are tight even in the constant and provide a precise dependence of communication complexity on the probabilities of error of two types.