A Set of Rules for Model Validation

TL;DR

This paper presents a comprehensive set of general rules to guide practitioners in validating data-driven models, aiming to improve reliability, transparency, and comparability of validation results.

Contribution

It introduces a standardized set of validation rules to assist practitioners in designing, reporting, and discussing model validation strategies effectively.

Findings

Rules promote transparent and reliable validation practices

Enhance comparability of validation results across studies

Help identify limitations in validation strategies

Abstract

The validation of a data-driven model is the process of assessing the model's ability to generalize to new, unseen data in the population of interest. This paper proposes a set of general rules for model validation. These rules are designed to help practitioners create reliable validation plans and report their results transparently. While no validation scheme is flawless, these rules can help practitioners ensure their strategy is sufficient for practical use, openly discuss any limitations of their validation strategy, and report clear, comparable performance metrics.