Bounds on data limits for all-to-all comparison from combinatorial designs

TL;DR

This paper investigates the minimum data storage requirements for all-to-all comparisons across multiple machines, establishing bounds and exploring combinatorial design structures to optimize data distribution efficiency.

Contribution

It introduces new bounds on the ATAC data limit, relates them to combinatorial parameters, and analyzes specific designs like transversal designs and projective Hjelmslev planes.

Findings

Established a lower bound on the ATAC data limit.

Identified conditions for equality in the bounds.

Explored data limits achievable with specific combinatorial designs.

Abstract

In situations where every item in a data set must be compared with every other item in the set, it may be desirable to store the data across a number of machines in such a way that any two data items are stored together on at least one machine. One way to evaluate the efficiency of such a distribution is by the largest fraction of the data it requires to be allocated to any one machine. The all-to-all comparison (ATAC) data limit for $m$ machines is a measure of the minimum of this value across all possible such distributions. In this paper we further the study of ATAC data limits. We observe relationships between them and the previously studied combinatorial parameters of fractional matching numbers and covering numbers. We also prove a lower bound on the ATAC data limit that improves on one of Hall, Kelly and Tian, and examine the special cases where equality in this bound is possible. Finally, we investigate the data limits achievable using various classes of combinatorial designs. In particular, we examine the cases of transversal designs and projective Hjelmslev planes.