On Ordinal Covering of Proposals Using Balanced Incomplete Block Designs

TL;DR

This paper introduces a new method using balanced incomplete block designs to assign proposals to referees in peer review systems, significantly reducing the number of referees needed while maintaining comprehensive pairwise coverage.

Contribution

The paper presents a novel approach that improves bounds on referee assignments, especially when the number of proposals per referee is between sqrt(n) and n/2, approaching theoretical optimality.

Findings

New bounds on referee assignments that are closer to the theoretical minimum.

The method achieves minimal referee coverage when k = sqrt(n) and k is a prime power.

The approach reduces the ratio of upper to lower bounds to within 3/2 for certain ranges of k.

Abstract

A frequently encountered problem in peer review systems is to facilitate pairwise comparisons of a given set of proposals by as few as referees as possible. In [8], it was shown that, if each referee is assigned to review k proposals then ceil{n(n-1)/k(k-1)} referees are necessary and ceil{n(2n-k)/k^2} referees are sufficient to cover all n(n-1)/2 pairs of n proposals. While the upper bound remains within a factor of 2 of the lower bound, it becomes relatively large for small values of k and the ratio of the upper bound to the lower bound is not less than 3/2 when 2 <= k <= n/2. In this paper, we show that, if sqrt(n) <= k <= n/2 then the upper and lower bounds can be made closer in that their ratio never exceeds 3/2. This is accomplished by a new method that assigns proposals to referees using a particular family of balanced incomplete block designs. Specifically, the new method uses ceil{n(n+k)/k^2} referees when n/k is a prime power, n divides k^2, and sqrt(n) <= k <= n/2. Comparing this new upper bound to the one given in [8] shows that the new upper bound approaches the lower bound as k tends to sqrt(n) whereas the upper bound in [8] approaches the lower bound as k tends to n. Therefore, the new method given here when combined together with the one in [8] provides an assignment whose upper bound referee complexity always remains within a factor of 3/2 of the lower bound when sqrt(n) <= k <= n, thereby improving upon the assignment described in [8]. Furthermore, the new method provides a minimal covering, i.e., it uses the minimum number of referees possible when k = sqrt(n) and k is a prime power.