Efficient candidate screening under multiple tests and implications for fairness

TL;DR

This paper extends screening models to multiple noisy tests, analyzing optimal employer policies and revealing fairness challenges when noise varies across groups in candidate evaluation.

Contribution

It introduces a multi-test screening framework with adaptive policies and highlights inherent fairness limitations across different candidate groups.

Findings

Optimal policies depend on test noise levels and adaptivity.

Impossibility results show fairness issues when noise differs across groups.

Multi-test models reveal complex trade-offs in candidate screening.

Abstract

When recruiting job candidates, employers rarely observe their underlying skill level directly. Instead, they must administer a series of interviews and/or collate other noisy signals in order to estimate the worker's skill. Traditional economics papers address screening models where employers access worker skill via a single noisy signal. In this paper, we extend this theoretical analysis to a multi-test setting, considering both Bernoulli and Gaussian models. We analyze the optimal employer policy both when the employer sets a fixed number of tests per candidate and when the employer can set a dynamic policy, assigning further tests adaptively based on results from the previous tests. To start, we characterize the optimal policy when employees constitute a single group, demonstrating some interesting trade-offs. Subsequently, we address the multi-group setting, demonstrating that when the noise levels vary across groups, a fundamental impossibility emerges whereby we cannot administer the same number of tests, subject candidates to the same decision rule, and yet realize the same outcomes in both groups.

Tables

Table 1. Table 1 : Confusion matrix for π greedy subscript 𝜋 greedy \pi_{\text{greedy}} assuming ϵ ≤ 1 / 4 italic-ϵ 1 4 \epsilon\leq 1/4 and ϵ ′ ≤ p ≤ 1 / 2 superscript italic-ϵ ′ 𝑝 1 2 \epsilon^{\prime}\leq p\leq 1/2 .

	General ${ϵ}^{\prime }$		When ${ϵ}^{\prime }=p$
	Skilled ($y_i=+1$)	Unskilled ($y_i=0$)	Skilled	Unskilled
accept	$\text{TPR}=\Theta \left(1-\frac{{ϵ}^{\prime }}{p}\left(1-\sigma \right)\right)$	$\text{FPR}=\Theta \left(ϵ\left(p-{ϵ}^{\prime }+{ϵ}^{\prime }\sigma \right)\right)$	$\Theta \left(\sigma \right)$	$\Theta \left(ϵp\sigma \right)$
reject	$\text{FNR}=\Theta \left(\frac{{ϵ}^{\prime }}{p}\left(1-\sigma \right)\right)$	$\text{TNR}=\Theta \left(1-ϵ\left(p-{ϵ}^{\prime }+{ϵ}^{\prime }\sigma \right)\right)$	$\Theta \left(1-\sigma \right)$	$\Theta \left(1-ϵp\sigma \right)$

Equations

$${\ell }_{\alpha }(b_1,b_2)=\alpha \mathbb{I}[b_1=0,b_2=1]+(1-\alpha )\mathbb{I}[b_1=1,b_2=0]$$

$$l_{\alpha }(\pi )=E[{\ell }_{\alpha }(y_i,\pi ({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau }))]$$

$$\mathrm{Pr}[\pi ({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau })=1|y_i=0]=\mathrm{Pr}[S_{\tau }\ge \theta |y_i=0]=\mathrm{Pr}[Z_{\tau }^{\eta }\ge \theta ].$$

$$\mathrm{Pr}[S_{\tau }=k|y_i=0]=\mathrm{Pr}[Z_{\tau }^{\eta }=k]=(\genfrac{}{}{0ex}{}{\tau }{k}){\eta }^k(1-\eta {)}^{\tau -k}.$$

$$\mathrm{Pr}[S_{\tau }=k|y_i=+1]=\mathrm{Pr}[Z_{\tau }^{\eta }=\tau -k]=(\genfrac{}{}{0ex}{}{\tau }{k}){\eta }^{\tau -k}(1-\eta {)}^k.$$

$${\theta }_{p,\alpha }^{\ast }=\arg \underset{\theta }{\min }{l}_{\alpha }(\theta )=\lceil \frac{\tau }{2}-\frac{\log (\frac{1}{p}-1)+\log (\frac{1}{\alpha }-1)}{2\log (1+\frac{2\sigma }{1-\sigma })}\rceil .$$

$$\arg \underset{\pi }{\min }$$

$$\frac{{\tau }_{\pi }}{\mathrm{Pr}[\pi ({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau })=1]}\le B$$

$$\underset{\tau }{\mathrm{minimize}}$$

$${\forall }_i\pi ({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau })=1\text{ iff }\mathrm{Pr}[y_i=+1|{\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau }]\ge 1-ϵ$$

$${\forall }_i\pi ({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau })=0\text{ iff }\mathrm{Pr}[y_i=+1|{\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau }]<{ϵ}^{\prime }$$

$${\pi }_{Greedy}({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,j})=\{\begin{array}{ll}0& \text{if }\mathrm{Pr}[y_i=+1|{\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,j}]<{ϵ}^{\prime }\\ 1& \text{if }\mathrm{Pr}[y_i=+1|{\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,j}]\ge 1-ϵ\\ \text{retest}& \text{else}\end{array}.$$

$${\pi }_{Greedy}({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau })=\{\begin{array}{ll}0& \text{if }{X}_{\tau }<\log \frac{{ϵ}^{\prime }}{1-{ϵ}^{\prime }}\\ 1& \text{if }{X}_{\tau }\ge \log (\frac{1-ϵ}{ϵ})\\ \text{retest}& \text{else}\end{array}$$

$$a=\lceil \frac{\log (\frac{(1-ϵ)(1-{ϵ}^{\prime })(1+\sigma )}{ϵ{ϵ}^{\prime }(1-\sigma )})}{\log (\frac{1+\sigma }{1-\sigma })}\rceil \gg \frac{1}{\sigma }\text{and }z=\lceil \frac{\log (\frac{p(1-{ϵ}^{\prime })(1+\sigma )}{{ϵ}^{\prime }(1-p)(1-\sigma )})}{\log (\frac{1+\sigma }{1-\sigma })}\rceil$$

$${\text{FPR}}_{\theta ,\tau }^{{\eta }_1}=\underset{{\eta }_1}{\mathrm{Pr}}[\pi ({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau })=1|y_i=0]<\underset{{\eta }_2}{\mathrm{Pr}}[\pi ({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau })=1|y_i=0]={\text{FPR}}_{\theta ,\tau }^{{\eta }_2},$$

$${\text{FNR}}_{\theta ,\tau }^{{\eta }_1}=\underset{{\eta }_1}{\mathrm{Pr}}[\pi ({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau })=0|y_i=+1]<\underset{{\eta }_2}{\mathrm{Pr}}[\pi ({\hat{y}}_{i,1},\dots ,{\hat{y}}_{i,\tau })=0|y_i=+1]={\text{FNR}}_{\theta ,\tau }^{{\eta }_2}.$$

$$\mathrm{Pr}\left[\left|\frac{1}{m}\sum _{i=1}^m{y}_i-\mathbb{E}[y_i]\right|\ge ϵ\right]\le 2e^{-m{ϵ}^2/2{\sigma }^2}.$$

$$\mathrm{Pr}\left[\left|\frac{1}{m}\sum _{i=1}^m{y}_i-p\right|\ge ϵ\right]\le 2e^{-2m{ϵ}^2}.$$

$$l_{\alpha }(\theta +1,\tau )-l_{\alpha }(\theta ,\tau )=-\alpha \mathrm{Pr}[y=0,S_{\tau }=\theta ]+(1-\alpha )\mathrm{Pr}[y=+1,S_{\tau }=\theta ]$$

$$=-\alpha \mathrm{Pr}[S_{\tau }=\theta |y=0]\mathrm{Pr}[y=0]+(1-\alpha )\mathrm{Pr}[S_{\tau }=\theta |y=+1]\mathrm{Pr}[y=+1]$$

$$l_{\frac{1}{2}}(\theta +1,\tau )-l_{\frac{1}{2}}(\theta ,\tau )=-(1-p)\alpha \mathrm{Pr}[S_{\tau }=\theta |y=0]+p(1-\alpha )\mathrm{Pr}[S_{\tau }=\theta |y=+1].$$

$$(1-p)\alpha \mathrm{Pr}[S_{\tau }=\theta |y=0]<p(1-\alpha )\mathrm{Pr}[S_{\tau }=\theta |y=+1]$$

$$\mathrm{Pr}[S_{\tau }=\theta |y=0]=(\genfrac{}{}{0ex}{}{\tau }{\theta })(\frac{1-\sigma }{2}{)}^{\theta }(\frac{1+\sigma }{2}{)}^{\tau -\theta }$$

$$\mathrm{Pr}[S_{\tau }=\theta |y=+1]=(\genfrac{}{}{0ex}{}{\tau }{\tau -\theta })(\frac{1-\sigma }{2}{)}^{\tau -\theta }(\frac{1+\sigma }{2}{)}^{\theta }$$

$$(1-p)\alpha \left(\genfrac{}{}{0ex}{}{\tau }{\theta }\right)(\frac{1-\sigma }{2}{)}^{\theta }(\frac{1+\sigma }{2}{)}^{\tau -\theta }<p(1-\alpha )\left(\genfrac{}{}{0ex}{}{\tau }{\tau -\theta }\right)(\frac{1-\sigma }{2}{)}^{\tau -\theta }(\frac{1+\sigma }{2}{)}^{\theta }.$$

$$(\frac{1-\sigma }{1+\sigma }{)}^{2\theta }<(\frac{1-\sigma }{1+\sigma }{)}^{\tau }(\frac{p}{1-p})(\frac{1-\alpha }{\alpha }).$$

$$2\theta \log (\frac{1-\sigma }{1+\sigma })<\tau \log (\frac{1-\sigma }{1+\sigma })+\log (\frac{p}{1-p})+\log (\frac{1-\alpha }{\alpha }).$$

$$\theta >\frac{\tau \log (\frac{1-\sigma }{1+\sigma })+\log (\frac{p}{1-p})+\log (\frac{1-\alpha }{\alpha })}{2\log (\frac{1-\sigma }{1+\sigma })}=\lceil \frac{1}{2}(\tau -\frac{\log (\frac{1}{p}-1)+\log (\frac{1}{\alpha }-1)}{\log (1+\frac{2\sigma }{1-\sigma })})\rceil$$

$$(1-p)\alpha \mathrm{Pr}[S_{\tau }=\theta |y=0]<p(1-\alpha )\mathrm{Pr}[S_{\tau }=\theta |y=+1],$$

$$\alpha (1-p)\mathrm{Pr}[S_{\tau }=\theta |y=0]>(1-\alpha )p\mathrm{Pr}[S_{\tau }=\theta |y=+1].$$

$$\mathrm{Pr}[\mathbb{E}[S_{\tau }]-S_{\tau }\ge ϵ|y=+1]=\mathrm{Pr}[\tau (\frac{1+\sigma }{2})-S_{\tau }\ge ϵ|y=+1]\le e^{-2{ϵ}^2\tau }<\delta .$$

$$\mathrm{Pr}[S_{\tau }-\mathbb{E}[S_{\tau }]\ge ϵ|y=0]=\mathrm{Pr}[S_{\tau }-\tau (\frac{1-\sigma }{2})\ge ϵ|y=0]\le e^{-2{ϵ}^2\tau }<\delta$$

Full text

1 Introduction

Consider an employer seeking to hire new employees. Clearly, the employer would like to hire the best employees for the task, but how will she know which are best fit? Typically, the employee will gather information on each candidate, including their education, work history, reference letters, and for many jobs, they will actively conduct interviews. Altogether, this information can be viewed as the signal available to the employer.

Suppose that candidates can be either skilled or unskilled. If the firm hires an “unskilled” candidate, it will incur a significant cost on account of lost productivity. For this reason, the employer would like to minimize the number of False Positive mistakes, instances where unskilled candidates are hired. On the other hand, the employer desires not to overspend on the hiring process, limiting the number of interviews per hired candidate (either on average, or absolutely). However, fewer interviews weakens the signal, causing the employer to make more mistakes. At the heart of our model is this inherent trade-off between the quality of the signal and the cost of obtaining the signal. This marks a departure from the classical economics literature, in which the signal is commonly regarded as a given.

Complicating matters, hiring efficiency is not the only desiderata at play. In society, employees belong to various demographic groups, and we may strive to design policies that are in some sense fair vis-a-vis group membership. While fairness can be an elusive notion, regulators must translate it to concrete rules and laws. In the United States, a body of anti-discrimination law dating to the Civil Rights act of 1964, subjects decisions that result in disparate outcomes (as delineated by race, age, gender, religion, etc) to extra scrutiny: employers must not only show that preference for some groups over others did not drive the decision (disparate treatment doctrine) but also justify that any observed disparities arise from a business necessity (disparate impact doctrine), whether or not those disparities were intentional.

In this paper, we seek to understand how a complex hiring process would interact with the requirements of fairness. We extend the theory on candidate screening and statistical discrimination, addressing the setting in which employers can subject employees to multiple tests, which we assume to be conditionally independent given the worker’s skill level and group membership. To build intuition, most of our analysis focuses on a Bernoulli model of both worker skill and screening. Additionally, we begin to extend the traditionally-studied Gaussian skill and screening models to the multi-test setting (Section 5).

Unlike the classical papers, in which an employer’s hiring policy is given by a simple thresholding rule, our setting requires greater care to derive the optimal employer policy. In our setting, we imagine that the employer wishes to minimize the number of tests performed subject to a constraint upper-bounding the false positive rate. We characterize the optimal policy in this case as a randomized threshold policy.

We also consider the setting in which employers can allocate tests dynamically, deciding after each result whether to (i) hire the candidate; (ii) reject the candidate and move on to the next one; or (iii) administer a subsequent test. In the Bernoulli case, the optimal policy consists of administering tests until each candidate’s posterior likelihood of being a high-skilled worker either dips below the prior or rises above a threshold determined by the tolerable false positive rate. We demonstrate that the analysis of this process can be reduced to a random walk over the log posterior odds and derive the solution via the corresponding Gambler’s ruin problem.

Finally, we consider the ramifications for fairness within our model when employees, despite possessing similarly-distributed skills, are evaluated with differing noise levels.

1.1 Related work

The classical economics literature on discrimination in employment can broadly be divided into two focuses. The taste-based discrimination model due to [3] models the market outcomes in a setting where employers express an explicit preference for hiring members of one group, acting as if an employee’s demographic membership provides utility. This preference for certain groups induces a sorting of employees from the disadvantaged group towards those employers who discriminate the least with wages ultimately determined by the marginal discriminator. Subsequently, [16] suggested a statistical mechanism by which similarly-skilled employees from different groups might experience differential outcomes: the comparative difficulty of screening from one group vs. another. Many subsequent works extend this analysis, typically focusing on Gaussian models of worker quality and conditionally-Gaussian test scores [2, 1]. These papers consider the setting where workers are assessed via a single test characterized by a group-dependent noise level. Our work is differentiated from these by considering richer mechanisms for acquiring signal.

In the more recent literature on fairness in machine learning, researchers often focus on binary classification, with employees characterized by a protected characteristic (group membership), and other (non-protected) covariates [15, 11, 12]. There, the predictor is presumably used to guide a consequential decision, such as allocating some economic good (loans, jobs, etc.) [6] or assessing some penalty (e.g. risk scores to guide bail decisions) [5]. Papers then focus on various interventions for ensuring accurate prediction subject to various constraints such as demographic parity (outcomes independent of group membership), blindness (model cannot observe group membership), and equalized false negative and/or false positive rates [9]. Several simple impossibility results preclude simultaneously satisfying several combinations of these parities [4, 5, 13]. More recently, a number of papers have drawn inspiration from economic modeling, extending the literature on fairness in classification to consider longer-term dynamics, equilibria, and the emergence of feedback loops [10, 9, 7]. Finally, [17] provide a survey of definitions from the algorithmic fairness literature.

2 The Bernoulli Model

We formalize our problem as follows. An employer accesses an infinite pool of candidates (indexed by $i\in\mathbb{N}^{+}$), each of which has some (hidden) skill level $y_{i}\in\{0,+1\}$, which denote unskilled and skilled, respectively. Underlying worker skill levels $y_{i}$ are sampled independently from a Bernoulli distribution with parameter $p$. An employer can access information about the $i$-th candidate through a sequence of $\tau$ tests, which are conditionally independent given $y_{i}$. Each test result, $\hat{y}_{i,j}\in\{0,+1\}$ disagrees with the ground truth skill with probability $\Pr[\hat{y}_{i,j}\neq y_{i}]=\frac{1-\sigma}{2}$, where $\sigma\in(0,1)$, i.e., $\hat{y}_{i,j}=y_{i}\mathbin{\oplus}Br(\frac{1-\sigma}{2})$111$\mathbin{\oplus}$ is the XOR operation between two binary random variables, and therefore $\hat{y}_{i,j}$ is also a random variable.. For convenience, we denote the noise level as $\eta=\frac{1-\sigma}{2}\in(0,\frac{1}{2})$. We say that a test result $\hat{y}_{i,j}$ is flipped if $\hat{y}_{i,j}\neq y_{i}$, and the number of flipped results for a given candidate is denoted by $Z_{\tau}^{\eta}$ is $Z_{\tau}^{\eta}=\sum_{j=1}^{\tau}\mathbb{I}(\hat{y}_{i,j}\neq y_{i})$, where $\mathbb{I}(\cdot)$ is the indicator function.

A selection criterion is a mapping between test results to actions: $\text{\tt Select}(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau_{i}})\in\{0,1\}$, where [math] means reject and $1$ means accept (hire). A policy $\pi$ sets the selection criteria based on $\sigma,p$ and other possible constraints such as probability to hire, error probability, etc. A randomized threshold policy is a policy $\pi$ with parameters $(\tau,\theta,r)$ such that $\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau_{i}})=1$ for $S_{\tau}:=\sum_{i=1}^{\tau}\hat{y}_{i,j}>\theta$, $\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau_{i}})=0$ for $S_{\tau}<\theta$, and for $S_{\tau}=\theta$ the probability that $\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau_{i}})=1$ is $r$. We call a policy $\pi$ a threshold policy if $r=1$. In a dynamic policy, rather than setting a fixed number of tests per candidate, the employer may decide after each test whether to accept, reject, or to perform an additional test, i.e., $\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau_{i}})\in\{0,1,\texttt{more}\}$. Note that for a dynamic policy, the number of tests $\tau$ is a random variable determined based on the tests’ outcomes. When designing a policy, one must carefully consider the balance between the following desiderata:

Minimize False Discovery Rate (FDR)—the fraction of unskilled workers among the accepted candidates, i.e., $\text{FDR}:=\Pr[y_{i}=0|\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau})=+1]$. 2. 2.

Minimize False Omission Rate (FOR)—the fraction of skilled workers among the rejected candidates, i.e., $\text{FOR}:=\Pr[y_{i}=+1|\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau})=0]$. 3. 3.

Minimize False Negatives (FN)—the amount of skilled workers that are classified as unskilled. 4. 4.

Minimize False Positives (FP)—the amount of unskilled workers that are classified as skilled. 5. 5.

Ratio of accept probability and the number of tests—the number of tests performed per candidate hired, using a parameter $B>1$, we have $\frac{\tau}{B}\leq\Pr[\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau})=+1]$.

For any fixed number of tests $\tau$, increasing the threshold $\theta$ of a threshold policy decreases FDR while increasing FOR.

Loss: To handle the trade-off between the false positives, (i.e., when an unskilled candidate is accepted) and false negatives (i.e., when a skilled candidate is rejected), we introduce an $\alpha$-loss, paramaterized by $\alpha\in[0,1]$ and defined as follows:

[TABLE]

where $\mathbb{I}[\cdot]$ is the indicator function and $b_{1},b_{2}\in\{0,1\}$. The expected loss of a policy $\pi$ is,

[TABLE]

where the expectation is over the type of the candidates $y_{i}$, the test results $\hat{y}_{i,j}$, and the decisions of $\pi$.

3 Analysis of the Bernoulli Model with One Group

To begin, we analyze this hiring model for a single group of candidates. The employer’s goal is to minimize the expected loss, $l_{\alpha}(\pi)$, while maintaining a given acceptance probability. For brevity, we relegate all proofs to the appendix.

3.1 The Simple Threshold Policy (Equal Number of Tests)

Consider the setting where the employer must subject all candidates to an equal number of tests $\tau$ and threshold $\theta$ (these parameters are chosen by the employer but thereafter constant across candidates). For a given threshold, we can relate the flip probability (error rate) of the test to the probability that a candidate is accepted as follows:

Recall that $\hat{y}_{i,j}=y_{i}\mathbin{\oplus}Br(\eta)$, $S_{\tau}=\sum_{j=1}^{\tau}\hat{y}_{i,j}$, that $Z^{\eta}_{\tau}=\sum_{t=1}^{\tau}\mathbb{I}(\hat{y}_{i,j}\neq y_{i})$, and that $\tau$ and $\theta$ are the only parameters of the threshold policy, $\pi$. Informally, $S_{\tau}$ is the number of passed tests and $Z^{\eta}_{\tau}$ is the number of flips (tests in error). The probability of hiring an unskilled candidate is given by:

[TABLE]

Since $Z_{\tau}^{\eta}$ is a binomial random variable with parameters $\tau$ and $\eta$, we can calculate this probability precisely as: $\Pr[\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau})=1|y_{i}=0]=\Pr[Z_{\tau}^{\eta}\geq\theta]=\sum_{k=\theta}^{\tau}{\tau\choose k}\eta^{k}(1-\eta)^{\tau-k}$, and the probability of rejecting a skilled candidate is the probability that they encounter more than $\tau-\theta$ flips, thus: $\Pr[\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau})=0|y_{i}=+1]=\Pr[S_{\tau}<\theta|y=+1]=\Pr[Z_{\tau}^{\eta}>\tau-\theta]=$

$\sum_{k=\tau-\theta+1}^{\tau}{\tau\choose k}\eta^{k}(1-\eta)^{\tau-k}.$ Similarly, given a candidate’s skill level, we can calculate the probability that they obtain exactly $k$ positive tests out of $\tau$, i.e,

[TABLE]

[TABLE]

Given these observations, we can now analyze the employer’s choices.

Optimal solution for any ratio $\alpha\in(0,1)$

The next theorem shows that for threshold policies, the expected loss $l_{\alpha}(\pi)=l_{\alpha}(\theta)$ is minimized at $\theta^{*}_{p,\alpha}$ such that $|\theta^{*}_{p,\alpha}-\tau/2|\leq\frac{\log(\frac{1}{p})+\log(\frac{1}{\alpha})}{2\log(1+\frac{2\sigma}{1-\sigma})}$.

Theorem 1**.**

The loss function $l_{\alpha}(\theta)$ is quasi-convex and a threshold of $\theta^{*}_{p,\alpha}=\left\lceil\frac{1}{2}\left(\tau-\frac{\log(\frac{1}{p}-1)+\log(\frac{1}{\alpha}-1)}{\log(1+\frac{2\sigma}{1-\sigma})}\right)\right\rceil$ minimizes loss for any values of $\alpha,p,\sigma\in(0,1)$. Namely,

[TABLE]

Next, we bound the number of tests required to guarantee that the probability of classification error by the majority decision rule (i.e., $\theta=\lceil\frac{\tau}{2}\rceil$) does not exceed a specified quantity $\delta$.

Theorem 2**.**

For every $\delta,p,\alpha\in(0,1)$, performing $\tau=\Omega(\frac{\alpha+p-2p\alpha}{\sigma^{2}}\ln(\frac{1}{\delta}))$ tests per candidate and using majority as a decision rule (i.e., $\theta=\tau/2$) guarantees $l_{\alpha}(\pi)\leq\delta$.

Equal cost for false positives and false negatives $(\alpha=\frac{1}{2})$

Consider the simple loss consisting of the classification error rate (false positives and false negatives count equally), expressed via our loss function by setting $\alpha=\frac{1}{2}$. When skilled and unskilled candidates occur with equal frequency, i.e., $p=1/2$, we can derive that the majority decision rule minimizes the classification error for any number of tests.

Corollary 3**.**

Assume $p=1/2$ and $\alpha=1/2$. For any number of tests $\tau$, the majority decision rule minimizes loss $l_{\alpha}$. Namely, $\arg\min_{\theta}l_{\frac{1}{2}}(\theta)=\lceil\frac{1}{2}\tau\rceil.$ In addition, for every $\delta\in(0,1)$, performing $\tau=\Omega(\frac{1}{\sigma^{2}}\ln(\frac{1}{\delta}))$ tests per candidate and using majority as a decision rule guarantees classification error with probability of at most $\delta$.

FDR minimization with limited number of tests per hire for balanced groups

Again, assuming balanced groups (i.e., $p=1/2$), suppose that an employer would like to minimize the false discovery rate, subject to the constraint of lower bounding the hiring probability. We can model this optimization problem by introducing a budget parameter $B>1$ to bound any predetermined (fixed) number of tests per hired candidate as follows:

[TABLE]

where $\tau_{\pi}$ is the number of tests $\pi$ performs. The following theorem shows that the optimal policy is a randomized threshold policy.

Theorem 4**.**

There exists a randomized threshold policy $\pi$ which is an optimal solution for (2).

3.2 The Dynamic Policy (Adaptively-Allocated Tests)

Recall that under a dynamic policy, the employer can decide after each test whether to accept, reject, or perform another test. In general, dynamic policies are more efficient than those that must set a fixed number of tests. To build intuition, consider a candidate that has passed $2$ out of $3$ tests. As seen above, under an optimally-constructed fixed-test policy, any candidate that fails a single test might be rejected. 222For example, if $B=18$ and $\eta=\frac{1}{3}$, the lowest false discovery rate is achieved by $\tau=\theta=3$. However, the posterior probability that this candidate is in fact skilled may still be greater than that of a fresh candidate sampled from the pool. Thus we can improve on the fixed-test policy by dynamically allocating more tests to candidates until their posterior odds either dip below the prior odds or rise above the threshold for hiring. The following theorem formalizes this notion that it is better to administer more tests to a candidate that passed the majority of previous tests than to start afresh with a new candidate:

Theorem 5**.**

For any $p,\sigma,\tau$, a candidate $i$ that passed $\theta>\frac{\tau}{2}$ out of $\tau$ tests is more likely to be a skilled than a freshly-sampled candidate $i^{\prime}$ for whom no test results are yet available, i.e., $\Pr[y_{i^{\prime}}=+1]=p<\Pr[y_{i}=+1|S_{\tau}=\theta]$.

Remark 6**.**

If $\theta<\frac{\tau}{2}$, the inequality would have been reversed.

The Greedy Policy

We now present a greedy algorithm that continues to test a candidate so long as the posterior probability that $y_{i}=+1$ is greater than $\epsilon^{\prime}$ and smaller than $1-\epsilon$, rejects a candidate whenever the posterior falls below $\epsilon^{\prime}$ (absent fairness concerns, employers will set $\epsilon^{\prime}=p$ for all groups), and accepts whenever the posterior rises above $1-\epsilon$. Given parameters $\epsilon,\epsilon^{\prime}>0$, we show that the greedy policy solves the optimization problem of minimizing the mean number of tests under these constraints, i.e.,

[TABLE]

Our analysis of this policy builds upon the observation that conditioned on a worker’s skill, the posterior log-odds after each test perform a one-dimensional random walk, starting with the prior log-odds $\log(\frac{p}{1-p})$ and moving, after each test result, either left (upon a failed test) or right (upon a passed test). When (as in our model) the probability of a flip are equal for skilled and unskilled candidates, our random walk has a fixed step size. Moreover, our random walk has absorbing barriers corresponding to (when $\epsilon^{\prime}=p$) falling below the prior log odds (on the left) and exceeding the hiring threshold (on the right). Owing to the fixed step size and absorbing barriers, our policy resembles the classic problem of Gambler’s ruin, in which a gambler wins or loses a unit of currency at each step, and loses when crossing a threshold on the left (going bankrupt) or on the right (bankrupting the opponent). We formalize the random walk as follows where $X_{j}$ is the position on the walk at time $j$:

$X_{0}$ is the prior log-odds of the candidate, i.e., $X_{0}=\log\frac{p}{1-p}$. 2. 2.

After each test result, $\hat{y}_{i,j}$ is observed, $X_{j}=X_{j-1}+(2\hat{y}_{i,j}-1)\cdot\log\left(\frac{\Pr[\hat{y}_{i,j}=+1|y_{i}=+1]}{\Pr[\hat{y}_{i,j}=+1|y_{i}=0]}\right)$.

Let $\pi_{Greedy}$ be the policy that accepts a candidate if $\Pr[y_{i}=+1|\hat{y}_{i,1},\dots,\hat{y}_{i,j}]\geq 1-\epsilon$, rejects if $\Pr[y_{i}=+1|\hat{y}_{i,1},\dots,\hat{y}_{i,j}]<\epsilon^{\prime}$, and otherwise conducts an additional test, i.e.,

[TABLE]

An employer will generally set the lower absorbing barrier to reject all candidates with posterior log odds less than $p$ since a fresh candidate from the pool is expected to be better. However, when noise levels differ across groups, we may prefer in the interest of fairness to set $\epsilon^{\prime}$ lower than $p$ for members of the noisier group, allowing us to equalize the frequency of false negatives across groups (see Section 4).

Lemma 7**.**

Let $\beta,\beta^{\prime}\in\mathbb{R}$ be the parameters that satisfy $\frac{\beta}{\beta+1}=1-\epsilon$ and $\frac{\beta^{\prime}}{\beta^{\prime}+1}=\epsilon^{\prime}$ (i.e., $\beta=\frac{1-\epsilon}{\epsilon}$and $\beta^{\prime}=\frac{\epsilon^{\prime}}{1-\epsilon^{\prime}}$). Then $X_{\tau}\geq\log\beta$ iff $\Pr[y_{i}=+1|\hat{y}_{i,1},\dots,\hat{y}_{i,\tau}]\geq 1-\epsilon$ (iff the candidate is accepted) and $X_{\tau}<\log\beta^{\prime}$ iff $\Pr[y_{i}=+1|\hat{y}_{i,1},\dots,\hat{y}_{i,\tau}]<\epsilon^{\prime}$ (iff the candidate is rejected).

Corollary 8**.**

The policy $\pi_{Greedy}$ can be described as follows.

[TABLE]

We use the following parameters in the next theorems:

[TABLE]

Theorem 9** (Expected number of tests per type).**

The expected number of tests until a decision (namely accept or reject) for skilled candidates is $\mathbb{E}[\tau_{s}]=\frac{1}{\sigma}\left(a\cdot\frac{1-(\frac{1-\sigma}{1+\sigma})^{z}}{1-(\frac{1-\sigma}{1+\sigma})^{a}}-z\right)\approx\frac{2a}{1+\sigma}-\frac{z}{\sigma}$ and $\mathbb{E}[\tau_{u}]=\frac{1}{\sigma}\left(z-a\cdot\frac{1-(\frac{1+\sigma}{1-\sigma})^{z}}{1-(\frac{1+\sigma}{1-\sigma})^{a}}\right)\approx\frac{z}{\sigma}$ for unskilled candidates.

For the probabilities of the candidates to be accepted or rejected, conditioned on their true skill level, we present the results in a form of confusion matrix in Table 1.

Theorem 10**.**

The expected number of tests until deciding whether to accept or reject a candidate is $\mathbb{E}[\tau|{\pi(y_{i,\tau})\in\{0,1\}}]\approx\frac{ap}{\sigma}$, where $a\gg\frac{1}{\sigma}$.

4 Fairness Considerations in the Two-Group Setting

Two Groups—Threshold Policies

We now discuss the effects of a threshold policy when candidates belong to two groups, $G_{1}$ and $G_{2}$ whose skill level is distributed identically, but whose tests are characterized by different noise levels. Without loss of generality, we assume that $\eta_{1}<\eta_{2}$, where $\eta_{i}$ is the probability that a test result of a candidate from $G_{i}$ is different from his skill level. To begin, we note the fundamental irreconcilability of equalizing either the false positive or the false negative rates across groups with subjecting candidates to the same policy.

Theorem 11** (Impossibility result).**

When noise levels differ between two groups with identical skill level distribution, a single Threshold Policy $\pi$ (with the same number of tests $\tau$ and the same threshold $\theta$ for both groups) cannot have equality in either the false negative rates or in the false positive rates across the groups. Particularly, there is a higher false positive rate in the noisier group, as an unskilled candidate from $G_{2}$ is more likely to be accepted by the threshold policy than an unskilled candidate from $G_{1}$:

[TABLE]

and also a higher false negative rate, as a skilled candidate from $G_{2}$ is more likely to be rejected than a skilled candidate from $G_{1}$:

[TABLE]

**Connection to Economics Literature **Aigner and Cain [1] discuss a similar case under a Gaussian screening model where the variance (noise level) of the single test differs across the two groups. Similarly, they note that qualified candidates fare worse in the noisy group but that unqualified candidates fare better in the noisier group. Our work differs from theirs in that we consider the effect of multiple tests and the ability to optimize over the number of tests.

Two Groups–Dynamic policy

We now consider the (dynamic) hiring policy in the setting when employees belong to two groups, $G_{1}$ and $G_{2}$ with identically-distributed skills but different noise levels $\eta_{1}<\eta_{2}$. We note that there are two ingredients that explain the differences among the groups: (i) The step size, $\log\left(\frac{\Pr[\hat{y}_{i,j}=+1|y_{i}=+1]}{\Pr[\hat{y}_{i,j}=+1|y_{i}=0]}\right)=\log\left(\frac{1-\eta}{\eta}\right)$ of $G_{2}$ (the noisier group) is smaller than the step size of $G_{1}$. Thus these candidates must typically pass more tests before they are accepted; and (ii) Skilled candidates in group $G_{2}$ exhibit less drift to the right (they have a higher probability of failing a test). Consequently, when an employer (rationally) sets $\epsilon^{\prime}=p$ for all groups, a skilled candidate from $G_{2}$ is more likely to be fail a test in step $1$, at which point the dynamic policy summarily rejects them. These two facts explain both the higher false negative rates for $G_{2}$ and the longer expected duration until acceptance. By setting $\epsilon^{\prime}<p$ for members of the noisier group, we can equalize false negative rates. Precisely, setting $\epsilon^{\prime}=\frac{\eta_{1}}{\eta_{2}}p$ achieves the desired parity. The cost of this intervention is that it requires more tests for candidates from the noisier group. Here, our random walk analysis can be leveraged to determine exactly how many more. Once again, we cannot provide equality across the groups in all desired ways—the same acceptance criterion, the same expected number of tests, and the same false negative rates between groups—with the noise differs across groups.

5 Gaussian Worker Screening Model

In this section, we work out the analytic solutions for the conditional expectation of worker qualities given a series of conditionally independent tests $Y_{1},...Y_{n}$ s.t. $\forall i,j$, $Y_{i}\perp Y_{j}|Q$. We assume that the worker quality $Q$ normally distributed with mean $\mu_{Q}$ and variance $\sigma_{Q}^{2}$, so instead of binary skill level we have continuous quality of candidates. Conditioned on $Q=q$, each test is generated according to the structural equation $y_{i}=q+\eta$, where $\eta$ is a normally distributed noise term with mean [math] and variance $\sigma_{\eta}^{2}$. Equivalently, we can say that the conditional distribution for each test $P(Y|Q=q)$ is Gaussian with mean $q$ and variance $\sigma_{\eta}^{2}$. We refer the reader to Appendix B for further details.

We show that we can equalize conditional variance between the two groups by giving more interviews to to noisier group, and that it yields the same conditional expectations.

Theorem 12**.**

For two groups, $G_{1},G_{2}$ with the same worker quality $Q$, that differ only in the variance of their noise $\sigma_{\eta_{1}}^{2}<\sigma_{\eta_{2}}^{2}$, the variance can be equalized by using $n_{2}=\frac{\sigma_{\eta_{2}}^{2}}{\sigma_{\eta_{1}}^{2}}n_{1}$ interviews (or tests) for $G_{2}$, where $n_{1}$ is the number of interviews for each candidate from $G_{1}$.

Theorem 13**.**

When equalizing conditional variances between $G_{1},G_{2}$ by using $n_{2}=\frac{\sigma_{\eta_{2}}^{2}}{\sigma_{\eta_{1}}^{2}}n_{1}$, we get the same conditional expectations, $\mathbbm{E}_{\eta_{1}}[Q|Y_{1},...,Y_{n_{1}}]=\mathbbm{E}_{\eta_{2}}[Q|Y_{1},...,Y_{n_{2}}]$.

6 Unsupervised Parameter Estimation

Now, under the assumption of realizable case, we explain how one can estimate the parameters $p$ and $\sigma$ given tests results from a homogeneous population. Surprisingly, we discover that parameter recovery in this model does not require any ground truth labels indicating whether an employee is skilled or unskilled. We use Hoeffding’s inequality to bound the absolute difference between the estimated parameters and the true parameters by choosing $\delta$ as the wanted upper bound and solving for the number of samples or $\epsilon$.

Lemma 14** (Hoeffding’s inequality).**

Let $y_{1},\dots,y_{m}$ be $\sigma^{2}-$sub–gaussian random variables. Then, for any $\epsilon>0$,

[TABLE]

If $y_{1},\dots,y_{m}$ are Bernoulli random variables with parameter $p$,

[TABLE]

We start by estimating $\sigma$ and then use it to derive an estimate for $p$. The estimated parameters are denoted by $\hat{\sigma}$ and $\hat{p}$. Notice that in order to have any information regarding the true value of $\sigma$, we need to have candidates with at least two tests. Hence, from now on we assume exactly that, i.e., $\forall_{i}\pi_{\text{Greedy}}(\hat{y}_{i,1})=more$ for dynamic policies and $\tau\geq 2$ for fixed number of tests policies.

Now, in both policies we have showed that the optimal rule is to reject candidates that fail their first test. Therefore inconsistencies between the first two tests are seen only in cases where $\hat{y}_{i,1}=1,\hat{y}_{i,2}=0$.

Let $c$ be the number of inconsistencies in the first two tests, i.e., $c=|\{(\hat{y}_{i,1},\hat{y}_{i,2}):y_{i,1}\neq y_{i,2}\}|$, and let $m$ be the number of candidates with at least two tests. Since $c$ is generated by sampling $m$ times, the distribution $Br((\frac{1+\sigma}{2})(\frac{1-\sigma}{2}))=Br(\frac{1-\sigma^{2}}{4})$ and we can estimate $\sigma$ as stated in the next theorem:

Theorem 15**.**

If we have results from $m\geq\frac{1}{2\epsilon^{2}}\ln\frac{2}{\delta}$ candidates, by using $\hat{\sigma}=\sqrt{1-4\frac{c}{m}}$, then with probability $1-\delta$ we have that $|\hat{\sigma}-\sigma|\leq\epsilon$.

Having an estimation of the parameter $\hat{\sigma}$, we can calculate the estimated $p$ as follows: Let $p_{\hat{y}_{*,1}=1}:=\frac{\sum_{i}\mathbb{I}(\hat{y}_{i,1}=1)}{m}$ be the percentage of positive first tests. Since this number is generated by the distribution $Br(\frac{1}{2}(p(1+\sigma)+(1-p)(1-\sigma)))=Br(\frac{1}{2}+(2p-1)\frac{\sigma}{2})$, we can estimate $\hat{p}$ using the estimated value of $\hat{\sigma}$.

Theorem 16**.**

If we have results from $m\geq\frac{1}{2\epsilon^{2}}\ln\frac{2}{\delta}$ candidates, by using $\hat{p}=\frac{2(p_{y_{*,1}=1}-1)+\hat{\sigma}}{\hat{{\sigma}}}$, we get that with probability $1-\delta$ we have that $|\hat{p}-p|\leq 2\epsilon$.

Under the Gaussian screening model, the parameter estimation is also straightforward (assuming realizability) without access to the true skill level of the employees. We start by looking at a single candidate, $i$. Each of his test results, $\hat{y}_{i,j}$ is generated from a conditional distribution $P(Y_{i}|Q_{i}=q_{i})$ which is a Gaussian with mean $q_{i}$ and variance $\sigma_{\eta}^{2}$. Since this variance is common among all the candidates, we can simply average the estimated variance of every candidate to get an approximation for $\sigma_{\eta}^{2}$. Suppose $\hat{y}_{i,1},\dots,\hat{y}_{i,n}$ is a sequence of $n$ i.i.d tests of candidate $i$, and let $\boldsymbol{y_{i}}=\frac{1}{n}\sum_{j=1}^{n}y_{i,j}$ be the empirical mean of candidate $i$’s tests.

The following theorem is a result from Hoeffding’s Inequality, in which we use to bound the error of our estimated parameters.

Theorem 17**.**

By using the following as estimators for Gaussian parameters $\hat{\mu}_{Q}=\frac{1}{m}\sum_{i=1}^{m}\boldsymbol{y_{i}}$, $\hat{\sigma}_{\eta}^{2}=\frac{1}{m}\sum_{i=1}^{m}\frac{1}{n}\sum_{j=1}^{n}(y_{i,j}-\boldsymbol{y_{i}})^{2}$ and $\hat{\sigma}_{Q}^{2}=\frac{1}{m}\sum_{i=1}^{m}(\hat{\mu}_{Q}-\boldsymbol{y_{i}})^{2}$ (notice that $\mathbb{E}[\hat{\sigma}_{\eta}^{2}]={\sigma}_{\eta}^{2}$ and $\mathbb{E}[\hat{\sigma}_{Q}^{2}]={\sigma}_{Q}^{2}$), the difference between each parameter and it’s estimator is bounded by $O(\sqrt{\frac{1}{m}\ln(\frac{1}{\delta})})$.

7 Discussion and Future Work

Consider two groups with identically-distributed skills and characterized by different noise levels in screening. Our results demonstrate that if a regulatory body (e.g., policymakers or a regulator) insists on the same number of tests and the same decision rule for both groups, this would yield higher false positive rates in any threshold policy. As a result, hired candidates from the noisier group would suffer higher rates of firing. In turn, this might lead employers to erroneously conclude that this group’s skill level is lower than it actually is. This paper presents a policy that handles this problem by minimizing the false positive rates of both groups, in the form of a greedy policy. Moreover the greedy policy is efficient, minimizing the expected number of tests per hire among all policies that achieve a specified false positive rate and continue testing every candidates that appear better than the a new one. However, the dynamic policy will still suffer (as does the simple threshold policy) from higher false negative rates for the noisier group, violating a notion of fairness dubbed equality of opportunity in the recent literature on fairness in machine learning [9]. We addressed this problem by modifying the greedy policy to reject candidate iff $\Pr[y_{i}=+1|\hat{y}_{i,1}\dots\hat{y}_{i,\tau}]<\epsilon^{\prime}$ by setting $\epsilon^{\prime}<p$. Our greedy policy can be made forgiving and equalize false negative rates across groups. In future work, we plan to explore extensions to the Gaussian model.

Appendix A Proofs

A.1 Proofs from Section 3

Proof of Theorem 1.

To prove the theorem, we show that the loss function $l_{\alpha}(\tau,\theta)$, as a function of $\theta$ is quasi-convex and achieves its minimum value at $\left\lceil\frac{1}{2}(\tau-\frac{\log(\frac{1}{p}-1)+\log(\frac{1}{\alpha}-1)}{\log(1+\frac{2\sigma}{1-\sigma})})\right\rceil$.

Namely, we show that the loss is monotone increasing for $\left\lceil\frac{1}{2}(\tau-\frac{\log(\frac{1}{p}-1)+\log(\frac{1}{\alpha}-1)}{\log(1+\frac{2\sigma}{1-\sigma})})\right\rceil\leq\theta\leq\tau-1$, i.e., increasing $\theta$ increases the loss: $l_{\alpha}(\theta)<l_{\alpha}(\theta+1)$.

Similarly, we show that for $1\leq\theta\leq\left\lceil\frac{1}{2}(\tau-\frac{\log(\frac{1}{p}-1)+\log(\frac{1}{\alpha}-1)}{\log(1+\frac{2\sigma}{1-\sigma})})\right\rceil$, we have $l_{\alpha}(\theta)<l_{\alpha}(\theta-1)$.

Indeed,

[TABLE]

[TABLE]

Since $\Pr[y=0]=1-p$ and $\Pr[y=+1]=p$, we have

[TABLE]

The above expression is positive iff

[TABLE]

Since $\Pr[S_{\tau}=\theta|y=0]$ is the probability of exactly $\theta$ flips, and $\Pr[S_{\tau}=\theta|y=+1]$ is the probability of exactly $\tau-\theta$ flips, we can calculate those probabilities as follows:

[TABLE]

[TABLE]

Substituting expression in (3), we get

[TABLE]

Rearranging, we get

[TABLE]

Applying $\log$ on both sides gets us

[TABLE]

Solving for $\theta$, we find that the inequality holds if

[TABLE]

For $\theta\geq\left\lceil\frac{1}{2}(\tau-\frac{\log(\frac{1}{p}-1)+log(\frac{1}{\alpha}-1)}{\log(1+\frac{2\sigma}{1-\sigma})})\right\rceil$, we have

[TABLE]

and for $\theta\leq\left\lceil\frac{1}{2}(\tau-\frac{\log(\frac{1}{p}-1)+\log(\frac{1}{\alpha}-1)}{\log(1+\frac{2\sigma}{1-\sigma})})\right\rceil$, we have

[TABLE]

This implies that the maximum is $\theta^{*}_{p,\alpha}=\left\lceil\frac{1}{2}(\tau-\frac{\log(\frac{1}{p}-1)+\log(\frac{1}{\alpha}-1)}{\log(1+\frac{2\sigma}{1-\sigma})})\right\rceil$.

∎

Proof of Theorem 2.

We start with a skilled candidate. The expected number of tests that a skilled candidate passes is $\mathbb{E}[S_{\tau}|y=+1]=\tau(\frac{1+\sigma}{2})>\frac{\tau}{2}$.

By using Hoeffding’s inequality for Bernoulli distributions, for every $\epsilon>0$,

[TABLE]

Choosing $\epsilon=\frac{\sigma}{2}$ yields $S_{\tau}\leq\frac{\tau}{2}<\lceil\frac{\tau}{2}\rceil$ (as $\tau$ is odd), which holds iff a majority threshold policy would predict that this is an unskilled candidate (false negative). Solving for $\tau$, we get $\tau>\frac{1}{\sigma^{2}}\ln(\frac{1}{\delta})$.

We now repeat the process for an unskilled candidate. The expected number of tests that an unskilled candidate passes is $\mathbb{E}[S_{\tau}|y=0]=\tau(\frac{1-\sigma}{2})<\frac{\tau}{2}$.

By using Hoeffding’s inequality again, we have

[TABLE]

Choosing $\epsilon=\frac{\sigma}{2}$ yields $S_{\tau}>\frac{\tau}{2}$, which holds iff a majority threshold falsely predicts that this is a skilled candidate (false positive). Solving for $\tau$ again, we get $\tau>\frac{1}{\sigma^{2}}\ln(\frac{1}{\delta})$.

Overall, $\tau>\frac{\alpha(1-p)}{\sigma^{2}}\ln(\frac{1}{\delta})+\frac{p(1-\alpha)}{\sigma^{2}}\ln(\frac{1}{\delta})=\Omega(\frac{\alpha+p-2p\alpha}{\sigma^{2}}\ln(\frac{1}{\delta}))$ ∎

Proof of Theorem 4.

Let $\pi^{\prime}$ be any optimal policy for (2) (not necessarily threshold) with a fixed number of tests, $\tau$. We will show, in two steps, how to transform it into an optimal randomized threshold policy. The first step is to symmetrize $\pi^{\prime}$. Let $r_{k}=\Pr[\pi(\hat{y})=1|S_{\tau}=k]$. Define a policy $\pi^{\prime\prime}$, which performs $\tau$ tests, and accepts with probability $r_{k}$ where $k=S_{\tau}$. Clearly, both $\pi^{\prime}$ and $\pi^{\prime\prime}$ have the same accept probability. In addition, since condition on $S_{\tau}=k$, any sequence of outcomes is equally likely. Furthermore, and the probability that $y=1$ given any sequence of outcomes with $S_{\tau}=k$, is identical. (Technically, $S_{\tau}$ is a sufficient statistics.) This implies that the false discovery rate is also unchanged.

This yields that $\pi$ with the randomization vector $r$ is also optimal.

The second step is to suppose—for sake of contradiction—that $\pi^{\prime\prime}$ is not a randomized threshold policy. We will show that we can improve the FDR of $\pi^{\prime\prime}$ while keeping the probability of acceptance unchanged. This will contradict the hypothesis that $\pi^{\prime}$ is optimal.

If $\pi^{\prime\prime}$ is not a randomized threshold policy, then there is no $\theta$ and $k$, such that

[TABLE]

Now, let $k$ be the minimal value such that $r_{k}>0$ and let $0<i<\tau-k$ be the minimal value for which $0<r_{k+i}<1$. Clearly, the FDR is lower at $S_{\tau}=k+i$ than at $S_{\tau}=k$. Intuitively, we can shift some probability mass, $\epsilon_{k}>0$ from $r_{k}$ to $r_{k+i}$ in a way that maintains the acceptance probability of $\pi$ and decreases the false positive rates.

Let $\epsilon_{k+i}>0$ be such that $\epsilon_{k}\cdot r_{k}=\epsilon_{k+i}\cdot r_{k+i}$. Let $r^{\prime}$ be a modified randomization vector for $\pi$ such that $r_{k}^{\prime}=r_{k}(1-\epsilon_{k}),r_{k+i}^{\prime}=r_{k+i}(1+\epsilon_{k+i})$ and for every $l\notin\{k,k+i\}$ $r_{l}^{\prime}=r_{l}$. Since $\Pr[\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau})=1]=\sum_{l=1}^{\tau}r_{l}=\sum_{l\notin\{k,k+i\}}r_{l}+r_{k}^{\prime}+r_{k+i}^{\prime}$, the acceptance probability remains the same. As for the false discovery rate, since $\Pr[y_{i}=0|S_{\tau}=k+i]<\Pr[y_{i}=0|S_{\tau}=k]$, $\Pr[S_{\tau}=k+i]$ is higher with $r^{\prime}$ than with $r$, $\Pr[S_{\tau}=k]$ is lower with $r^{\prime}$ than with $r$ and for any $l\notin\{k,k+i\}$, $\Pr[S_{\tau}=l]$ with $r^{\prime}$ is the same as with $r$, the false discovery rate with $r^{\prime}$ is lower, which contradicts the optimality of $\pi$ with $r$ as the randomization vector. ∎

Proof of Theorem 5.

Using Bayes’ theorem, the conditional probability can be decomposed as

[TABLE]

[TABLE]

Since $\tau-\theta<\theta$ and ${\tau\choose\theta}={\tau\choose\tau-\theta}$, we get

[TABLE]

Since $(\frac{1+\sigma}{1-\sigma})>1$ it holds that $(\frac{1+\sigma}{1-\sigma})^{2\theta-\tau}>1$,

[TABLE]

So,

[TABLE]

And finally,

[TABLE]

∎

Proof of Lemma 7.

Let $S^{\prime}_{\tau}=\sum_{j=1}^{\tau}(2\hat{y}_{i,j}-1)$, and let $s_{\tau}\in\{-\tau,\dots,\tau\}$ be any of the possible values of $S^{\prime}_{\tau}$. Note that

[TABLE]

Since the $\hat{y}_{i,j}$ are i.i.d., we have

[TABLE]

Since

[TABLE]

we have

[TABLE]

Since

[TABLE]

and

[TABLE]

assigning $\Pr[y_{i}=0]=1-p$ and $\Pr[y_{i}=1]=p$, we get

[TABLE]

Applying (5) in (4) and adding $X_{\tau}\geq\log\beta$ gives us

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Applying (5) in (4) and adding $X_{\tau}<\log\beta^{\prime}$ gives us

[TABLE]

Hence

[TABLE]

∎

Proof of Theorem 9.

First recall that given a skilled candidate, for every test $j$,

[TABLE]

[TABLE]

Hence

[TABLE]

The lower absorbing barrier is reached when a candidate’s posterior skill level is lower than the prior of the skill level, i.e.,

[TABLE]

and the starting point is just one step away from the lower absorbing barrier:

[TABLE]

According to Corollary 8, the upper absorbing barrier is in

[TABLE]

To derive the results for the expected duration of the random walk for skilled and unskilled candidates, we shift the locations of the absorbing points so that the lower barrier would be in 0 and also divide them by a step size (so now we have that every step is of size $1$). The new upper absorbing barrier is at

[TABLE]

And we also shift the starting point:

[TABLE]

As stated in [8], the expected duration of a random walk with absorbing barriers of [math] and $a$ from $z=1$ is (equation 3.4, chapter XIV [page 348]):

[TABLE]

Hence,

[TABLE]

As for unskilled candidates, the absorbing points and the starting point are the same, the only difference is that

[TABLE]

and

[TABLE]

Therefore,

[TABLE]

and we deduce

[TABLE]

∎

Deviations for the confusion matrix (Table 1).

We split the claim in the confusion matrix (Table 1) into two parts. First, using equation (2.4) from chapter XIV [page 345] in [8], we get

[TABLE]

and

[TABLE]

The second part follows from the fact the gambler’s ruin must end in case of absorbing barriers.

[TABLE]

[TABLE]

Where $\mu:=\frac{\epsilon^{\prime}(1-\sigma)}{(1-\epsilon^{\prime})(1+\sigma)}$. For $\epsilon\leq 1/4$ and $p<1/2$ we get $0\leq\mu\leq 1/3$ and $\mu=\Theta(\epsilon^{\prime}(1-\sigma))$, therefore

[TABLE]

Hence $\text{FNR}=\Theta(\frac{\epsilon^{\prime}}{p}(1-\sigma))$.

[TABLE]

[TABLE]

Hence $\text{TNR}=\Theta\left(1-\epsilon(p-\epsilon^{\prime}+\epsilon^{\prime}\sigma)\right)$. ∎

Proof of Theorem 10.

[TABLE]

[TABLE]

[TABLE]

∎

A.2 Proofs from Section 4

The next lemma aids in the proof of Theorem (11).

Lemma 18**.**

Let $Z_{n}^{\eta}$ be a Binomial random variable with parameters $n\in\mathbb{N}$ and $\eta\in(0,1)$. Given a number of successes, $k\in\{0,\dots,n\}$, we know that the probability mass function of $Z_{n}^{\eta}$ is $f_{k}(\eta):=\Pr[Z_{n}^{\eta}=k]={n\choose k}\eta^{k}(1-\eta)^{n-k}$. Let $\mathcal{L}(\eta|k)$ be the likelihood function of the event $Z_{n}^{\eta}=k$. Then the maximum likelihood of $f_{k}(\eta)$ is $\eta=\frac{k}{n}$. I.e.,

[TABLE]

Proof of Lemma 18.

We notice that ${n\choose k}$ does not depend on $\eta$, thus

[TABLE]

The log-likelihood is particularly convenient for maximum likelihood estimation. Logarithms are strictly increasing functions, as a result, maximizing the likelihood is equivalent to maximizing the log-likelihood, i.e.,

[TABLE]

Differentiating (with respect to $\eta$) and comparing to zero we get

[TABLE]

And after refactoring,

[TABLE]

The function $\ln(f_{k}(\eta))$ is a strictly concave as its second derivative is negative,

[TABLE]

And since the derivative of a strictly concave function is zero at $\frac{k}{n}$, then $\hat{\eta}=\frac{k}{n}$ is a global maximum. Therefore, $\hat{\eta}=\frac{k}{n}$ obtains absolute maximum in $f_{k}(\eta)$. ∎

Proof of Theorem 11.

Let $Z_{\tau}^{\eta_{i}}$ be a random variable that represents the number of flips out of a $\tau$-tests sequence with a noise level of $\eta_{i}$, i.e., $Z_{\tau}^{\eta_{i}}$ is the number of times when $y_{j}\neq y$ for $1\leq j\leq\tau$. We use $Z_{\tau}^{\eta_{i}}$ to express $\Pr[\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau})=1|y_{i}=0,\eta=\eta_{i}]$ as the probability that at least $\theta$ flips,

[TABLE]

and the probability of $\Pr[\pi(\hat{y}_{i,1},\dots,\hat{y}_{i,\tau})=1|q=+1,\eta=\eta_{i}]$ as at most $\tau-\theta$ flips, thus

[TABLE]

From Lemma (18) and since probability density function (pdf) are is monotone increasing, we derive that the pdf of $Z_{n}^{\eta_{2}}$ satisfies monotone likelihood ratio property over the pdf of $Z_{n}^{\eta_{1}}$. This implies that the pdf of $Z_{n}^{\eta_{2}}$ also has first-order stochastic dominance over $Z_{n}^{\eta_{1}}$ by Theorem 1.1 in [18]. From stochastic dominance, we can derive the desired inequalities

[TABLE]

and

[TABLE]

∎

Appendix B Gaussian Worker Screening Model Extension

In this extension, we characterize the conditional expectation, $\mathbbm{E}[Q|Y_{1},...,Y_{n}]$ and the conditional variance of $Q$ given the tests$Y_{i}$, i.e., $\text{Var}[Q|Y_{1},...,Y_{n}]$.

First, note that because $P(Q)$ is Gaussian, and the conditionals $P(Y_{i}|Q)$ are all Gaussian, the joint probability $P(Q,Y_{1},...,Y_{n})$ is a multivariate Gaussian. We work out the precise analytic forms for the mean and variance of the conditional $P(Q|Y_{1},...,Y_{n})$ in terms of the quality and noise variances ($\sigma_{Q}^{2}$ and $\sigma_{\eta}^{2}$) and the number of tests $n$.

To begin, we note the properties of the joint distribution over $P(Q,Y_{1},...,Y_{n})$. Owing to the generative process for our $Y_{i}$, all have mean $\mu_{Q}$, and thus the joint over the means is an $n+1$-dimensional vector $(\mu_{Q},...,\mu_{Q})$. The full $n+1\times n+1$ covariance matrix $\Sigma$ has the form

[TABLE]

where all off-diagonal entries $\Sigma_{ij}$ for $i\neq j$ have value $\sigma_{Q}^{2}$ and all diagonal entries $i=j\geq 2$ corresponding to the variance of tests $Y_{i}$ take value $\sigma_{Q}^{2}+\sigma_{\eta}^{2}$. The top-left entry corresponds to the variance of the test Q and thus has variance $\sigma_{Q}^{2}$.

We can now derive the equations for the conditional mean and conditional variance of $Q|Y_{1},...,Y_{n}$. To begin, note the following basic facts about deriving conditionals from multivariate Gaussians: to estimate the conditional of one set of variables, given another set $P(\mathbf{x}_{1}|\mathbf{x}_{2})$ we can segment our data matrix into those rows corresponding to the variables we don’t condition upon (here, just $Q$) and those we do (here, $Y_{1},..,Y_{n}$), expressing our covariance matrix in terms of the following submatrices:

[TABLE]

Here, $\Sigma_{11}\in\mathbbm{R}^{1\times 1}$, $\Sigma_{21}\in\mathbbm{R}^{n\times 1}$, $\Sigma_{21}\in\mathbbm{R}^{1\times n}$, and $\Sigma_{22}\in\mathbbm{R}^{n\times n}$.

The conditional expectation $\mathbbm{E}[Q|Y_{1},...,Y_{n}]$ is then expressed as

[TABLE]

and the conditional variance $Var[Q|Y_{1},...,Y_{n}]$ can be expressed as

[TABLE]

which should be familiar as the Schur complement of the $n\times n$ matrix $\Sigma_{22}$. Intuitively, this corresponds to inverting the full matrix $\Sigma$, deleting those rows and columns corresponding to observed variables, and then inverting the resulting ($1\times 1$) matrix back.

What remains is to show that for the particular covariance matrix that interests us (Equation 6), these expressions have simple analytically computable forms. Specifically we state the following simple theorems.

Theorem 19**.**

For jointly Gaussian variables $Q,Y_{1},...,Y_{n}$ characterized by the covariance matrix given in (Equation 6), the conditional expectation takes form

[TABLE]

Theorem 20**.**

For jointly Gaussian variables $Q,Y_{1},...,Y_{n}$ characterized by the covariance matrix given in (Equation 6), the conditional variance of $Q$ given the tests $Y_{i}$ takes form

[TABLE]

B.1 Proofs from Appendix B

Proof of Theorem 19.

The crucial step to apply Equation 7 to this data is to work out a simple expression for the inverse of the $n\times n$ submatrix $\Sigma_{22}$. We recall that this matrix is symmetrical with all diagonal entries equal to $\sigma_{Q}^{2}+\sigma_{\eta}^{2}$ and all off diagonals equal to $\sigma_{Q}^{2}$.

We call upon a lemma due to [14]. Which states that when $A$ is invertible and $B$ is a rank-1 matrix, the inverse of their sum takes the following form:

[TABLE]

We can decompose $\Sigma_{22}$ into such an $A$ and $B$ by defining $B$ to be the matrix that takes value $\sigma_{Q}^{2}$ everywhere and $A$ to be a diagonal matrix that takes values $\sigma_{\eta}^{2}$ (along the main diagonal). Thus $\Sigma_{22}^{-1}=(A+B)^{-1}$ and we can proceed by applying the lemma.

First we note that $A^{-1}$ is a diagonal matrix with all entries on the main diagonal set to $\frac{1}{\sigma_{\eta}^{2}}$. Then we note that $BA^{-1}$ is an $n\times n$ matrix with all entries set to $\frac{\sigma_{Q}^{2}}{\sigma_{\eta}^{2}}$. Thus, $\text{Trace}(BA^{-1})=\frac{n\sigma_{Q}^{2}}{\sigma_{\eta}^{2}}$.

The matrix $A^{-1}BA^{-1}$ has all entries equal to $\frac{\sigma_{Q}^{2}}{\sigma_{\eta}^{4}}$, and thus our desired inverse $\Sigma_{22}^{-1}$ can be expressed as follows:

[TABLE]

Thus each off-diagonal entry takes values

[TABLE]

and each on-diagonal entry has an additional $\frac{1}{\sigma_{\eta}^{2}}$ term that comes from $A^{-1}$.

[TABLE]

Now that we know the precise expression for all entries of $\Sigma_{22}^{-1}$, we can calculate the vector-matrix product $\Sigma_{12}\Sigma_{22}^{-1}$. Because every entry of $\Sigma_{12}$ takes value $\sigma_{Q}^{2}$, and because every column of $\Sigma_{22}^{-1}$ has the same $n$ values (just in different order), the product $\Sigma_{12}\Sigma_{22}^{-1}$ is an $n$ dimensional vector, where all $n$ values are equal:

[TABLE]

This expression for $\Sigma_{12}\Sigma_{22}^{-1}$, together with the definition of the conditional expectation of a multivariate Gaussian (Equation 7) concludes the proof.

∎

Proof of Theorem 20.

We can now produce the expression for $\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$. Because every entry of the $1\times n$ matrix $\Sigma_{12}\Sigma_{22}^{-1}$ takes value

[TABLE]

and because every entry in the $n\times 1$ matrix $\Sigma_{21}$ takes value $\sigma_{Q}^{2}$,

[TABLE]

The expression for the conditional variance $\text{Var}[Q|Y_{1},...,Y_{n}]$ follows:

[TABLE]

∎

B.2 Proofs

Proof of Theorem 12.

First, recall that

[TABLE]

Solving for $n_{2}$ in the equation $\text{Var}_{1}[Q|Y_{1},...,Y_{n_{1}}]=\text{Var}_{2}[Q|Y_{1},...,Y_{n_{2}}]$,

[TABLE]

we get

[TABLE]

and hence

[TABLE]

Extracting $n_{2}$, we find that $n_{2}=\frac{\sigma_{\eta_{2}}^{2}}{\sigma_{\eta_{1}}^{2}}n_{1}$. ∎

Proof of Theorem 13.

First, recall that

[TABLE]

Now,

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

∎