Can we trust the standardized mortality ratio? A formal analysis and evaluation based on axiomatic requirements

TL;DR

This paper rigorously analyzes the standardized mortality ratio (SMR) using axiomatic requirements, revealing significant limitations in its ability to reliably compare hospital performance due to violations of key properties.

Contribution

It introduces five axiomatic criteria for evaluating mortality measures and demonstrates that the SMR fails to meet most of these criteria under common standardization methods.

Findings

SMR satisfies only two axioms under external standardization.

SMR fails multiple axioms under internal standardization.

Variations in patient mix and mortality rates affect SMR validity.

Abstract

Background: The standardized mortality ratio (SMR) is often used to assess and compare hospital performance. While it has been recognized that hospitals may differ in their SMRs due to differences in patient composition, there is a lack of rigorous analysis of this and other - largely unrecognized - properties of the SMR. Methods: This paper proposes five axiomatic requirements for adequate standardized mortality measures: strict monotonicity, case-mix insensitivity, scale insensitivity, equivalence principle, and dominance principle. Given these axiomatic requirements, effects of variations in patient composition, hospital size, and actual and expected mortality rates on the SMR were examined using basic algebra and calculus. In this regard, we distinguished between standardization using expected mortality rates derived from a different dataset (external standardization) and standardization based on a dataset including the considered hospitals (internal standardization). Results: Under external standardization, the SMR fulfills the axiomatic requirements of strict monotonicity and scale insensitivity but violates the requirement of case-mix insensitivity, the equivalence principle, and the dominance principle. All axiomatic requirements not fulfilled under external standardization are also not fulfilled under internal standardization. In addition, the SMR under internal standardization is scale sensitive and violates the axiomatic requirement of strict monotonicity. Conclusions: The SMR fulfills only two (none) out of the five proposed axiomatic requirements under external (internal) standardization. Generally, the SMRs of hospitals are differently affected by variations in case mix and actual and expected mortality rates unless the hospitals are identical in these characteristics. These properties hamper valid assessment and comparison of hospital performance based on the SMR.