Concentration Inequalities for Exchangeable Tensors and Matrix-valued Data

TL;DR

This paper develops new concentration inequalities for structured exchangeable data, including tensors and matrices, providing sharper bounds and applications to multi-factor models and federated learning.

Contribution

It introduces novel Hoeffding and Bernstein bounds for exchangeable tensors and matrices, extending classical inequalities beyond independence.

Findings

Sharper concentration bounds for matrix arrays than previous methods

Theoretical predictions are validated by numerical experiments

New analytical tools for multi-factor response models and federated averaging

Abstract

We study concentration inequalities for structured weighted sums of random data, including (i) tensor inner products and (ii) sequential matrix sums. We are interested in tail bounds and concentration inequalities for those structured weighted sums under exchangeability, extending beyond the classical framework of independent terms. We develop Hoeffding and Bernstein bounds provided with structure-dependent exchangeability. Along the way, we recover known results in weighted sum of exchangeable random variables and i.i.d. sums of random matrices to the optimal constants. Notably, we develop a sharper concentration bound for combinatorial sum of matrix arrays than the results previously derived from Chatterjee's method of exchangeable pairs. For applications, the richer structures provide us with novel analytical tools for estimating the average effect of multi-factor response models and studying fixed-design sketching methods in federated averaging. We apply our results to these problems, and find that our theoretical predictions are corroborated by numerical evidence.