Formal and Informal Model Selection with Incomplete Data

TL;DR

This paper discusses the challenges of model selection with incomplete data, emphasizing the need to assess both fit to observed data and sensitivity to assumptions, illustrated through continuous and categorical examples.

Contribution

It introduces a framework for model assessment with incomplete data, focusing on fit and sensitivity analysis, which addresses limitations of traditional methods.

Findings

Model assessment should include fit to observed data and sensitivity to assumptions.

Incomplete data complicates direct model-data comparisons.

Sensitivity analysis helps evaluate the robustness of inferences.

Abstract

Model selection and assessment with incomplete data pose challenges in addition to the ones encountered with complete data. There are two main reasons for this. First, many models describe characteristics of the complete data, in spite of the fact that only an incomplete subset is observed. Direct comparison between model and data is then less than straightforward. Second, many commonly used models are more sensitive to assumptions than in the complete-data situation and some of their properties vanish when they are fitted to incomplete, unbalanced data. These and other issues are brought forward using two key examples, one of a continuous and one of a categorical nature. We argue that model assessment ought to consist of two parts: (i) assessment of a model's fit to the observed data and (ii) assessment of the sensitivity of inferences to unverifiable assumptions, that is, to how a model described the unobserved data given the observed ones.