Reliability of decisions based on tests: Fourier analysis of Boolean decision functions

TL;DR

This paper uses Fourier analysis of Boolean functions to evaluate the stability and reliability of decision functions in tests, advocating for weighted sum scores that align with test theory and exhibit desirable properties.

Contribution

It introduces Fourier analysis techniques to assess decision stability and influence of items, providing a theoretical foundation for using weighted sum scores in test decisions.

Findings

Weighted sum scores are stable and reliable for decision-making.

Fourier analysis identifies influential items affecting test decisions.

The approach aligns with test theory and enhances decision robustness.

Abstract

Items in a test are often used as a basis for making decisions and such tests are therefore required to have good psychometric properties, like unidimensionality. In many cases the sum score is used in combination with a threshold to decide between pass or fail, for instance. Here we consider whether such a decision function is appropriate, without a latent variable model, and which properties of a decision function are desirable. We consider reliability (stability) of the decision function, i.e., does the decision change upon perturbations, or changes in a fraction of the outcomes of the items (measurement error). We are concerned with questions of whether the sum score is the best way to aggregate the items, and if so why. We use ideas from test theory, social choice theory, graphical models, computer science and probability theory to answer these questions. We conclude that a weighted sum score has desirable properties that (i) fit with test theory and is observable (similar to a condition like conditional association), (ii) has the property that a decision is stable (reliable), and (iii) satisfies Rousseau's criterion that the input should match the decision. We use Fourier analysis of Boolean functions to investigate whether a decision function is stable and to figure out which (set of) items has proportionally too large an influence on the decision. To apply these techniques we invoke ideas from graphical models and use a pseudo-likelihood factorisation of the probability distribution.