Estimating psychometric functions from adaptive designs

TL;DR

This paper examines how adaptive experimental designs in psychometric studies influence the accuracy of estimating psychometric functions, highlighting small-sample bias issues and their implications for statistical inference.

Contribution

It provides an analytical and numerical investigation into the bias introduced by adaptive designs in psychometric function estimation, especially in small samples.

Findings

Adaptive designs preserve asymptotic estimator properties.

Small-sample bias affects slope parameter estimates.

Trade-off exists between sampling efficiency and bias reduction.

Abstract

An adaptive design adjusts dynamically as information is accrued and a consequence of applying an adaptive design is the potential for inducing small-sample bias in estimates. In psychometrics and psychophysics, a common class of studies investigate a subject's ability to perform a task as a function of the stimulus intensity, meaning the amount or clarity of the information supplied for the task. The relationship between the performance and intensity is represented by a psychometric function. Such experiments routinely apply adaptive designs, which use both previous intensities and performance to assign stimulus intensities, the strategy being to sample intensities where the information about the psychometric function is maximised. Similar schemes are often applied in drug trials to assign doses dynamically using doses and responses from earlier observations. The present paper investigates the influence of adaptation on statistical inference about the psychometric function focusing on estimation, considering both parametric and non-parametric estimation under both fixed and adaptive designs in schemes encompassing within subject independence as well as dependence through random effects. We study the scenarios analytically, focussing on a latent class model to derive results under random effects, and numerically through a simulation study. We show that while the asymptotic properties of estimators are preserved under adaptation, the adaptive nature of the design introduces small-sample bias, in particular in the slope parameter of the psychometric function. We argue that this poses a dilemma for a study applying an adaptive design in the form of a trade-off between more efficient sampling and the need to increase the number of samples to ameliorate small-sample bias.