Models for the assessment of treatment improvement: the ideal and the feasible

TL;DR

This paper examines the limitations of traditional statistical methods for comparing treatments, especially regarding stochastic dominance, and proposes feasible approaches to gather evidence for relaxed forms of treatment improvement.

Contribution

It highlights the limitations of classical methods for stochastic dominance and introduces feasible statistical evidence for relaxed dominance conditions.

Findings

Classical tests often cannot statistically prove stochastic dominance.

Feasible methods can gather evidence for approximate stochastic dominance.

Limitations of mean comparison under normal models are discussed.

Abstract

Comparisons of different treatments or production processes are the goals of a significant fraction of applied research. Unsurprisingly, two-sample problems play a main role in Statistics through natural questions such as `Is the the new treatment significantly better than the old?'. However, this is only partially answered by some of the usual statistical tools for this task. More importantly, often practitioners are not aware of the real meaning behind these statistical procedures. We analyze these troubles from the point of view of the order between distributions, the stochastic order, showing evidence of the limitations of the usual approaches, paying special attention to the classical comparison of means under the normal model. We discuss the unfeasibility of statistically proving stochastic dominance, but show that it is possible, instead, to gather statistical evidence to conclude that slightly relaxed versions of stochastic dominance hold.